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Overview 


Low-income preschool children face many risks to their social-emotional development that can af- 
fect their school experience and social outcomes for years to come. Although there are some promis- 
ing approaches to improving young children’s social-emotional competence, the evidence base is 
limited, particularly on the effectiveness of these approaches when operating at large scale. 

To help address this gap, the Head Start CARES demonstration evaluated the implementation and 
impacts of three interventions that included classroom-based social-emotional strategies, along with 
professional development (teacher training and coaching) and related supports for scaling up the 
program, in Head Start classrooms across the country. “The Incredible Years Teacher Training Pro- 
gram” trains teachers to create an organized classroom climate that supports children’s behavior 
regulation (by both the teacher and by the children themselves) and promotes positive teacher-child 
relationships. “Preschool PATHS” focuses on improving children’s social-emotional problem- 
solving skills, including defining problems and anticipating the consequences of different solutions. 
“Tools of the Mind — Play” focuses on training teachers to strengthen children’s ability to regulate 
their behavior by using adult-supported, “make believe” play and other activities. 

The demonstration was conducted with 17 Head Start grantees that varied by geographic location, 
organizational setting, and size, and generally represented the diversity of Head Start settings nation- 
ally. Centers within these 17 grantees were randomly assigned to one of the three interventions or to 
a “business as usuaf’ control group. This report describes the extent to which each strategy and the 
professional development supports were implemented as intended, and the degree to which teachers’ 
practices changed as a result. 


Key Findings 

Taken together, the following findings confirm that the Head Start CARES demonstration provided 
a fair test of large-scale implementation of the three interventions, thereby providing a sound basis 
for evaluating their impact on children and classrooms. 

• Teacher attendance at training sessions and the quality of training were generally strong across 
all grantees and interventions. The dosage (frequency and duration) and quality of coaching for 
teachers in the classroom were also generally high. 

• Despite challenges, teachers reported that they understood the interventions and were able to 
implement them. The Incredible Years and Preschool PATHS were easier for teachers to im- 
plement than was Tools of the Mind. 

• Classroom implementation varied somewhat across the three interventions in terms of fidelity to 
the original model, but each intervention was implemented with fidelity at or above a predeter- 
mined threshold rating of 3 on a 5 -point scale. Fidelity of classroom implementation improved 
over the course of the school year. 

• Compared with the control classrooms, each intervention improved the specific teacher practice 
that it was hypothesized to affect. The Incredible Years improved teachers’ classroom manage- 
ment, PATHS bolstered teachers’ social-emotional instruction, and Tools of the Mind increased 
teachers’ scaffolding of peer interactions and play, in which the teacher supports the children in 
their attempts to push beyond their current skill level. 

A separate report examines the impacts of the three interventions on teacher-child interactions, chil- 
dren’s preschool outcomes, and children’s kindergarten outcomes. 


v 
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Executive Summary 


Low-income preschool children face a number of risks that can cause their social-emotional 
development to lag behind that of their more affluent peers. ' This imbalance is of concern be- 
cause early social-emotional development is associated with better school readiness and peer 
relationships, which may contribute to a lower likelihood of high school dropout or lower rates 
of delinquency.^ It has been suggested that social-emotional skills may allow children to engage 
more fully in the classroom, setting them up for even better classroom participation and poten- 
tially better academic outcomes. ^Direct intervention to close the gap in social-emotional devel- 
opment between low- and higher-income children is, therefore, thought to be one way to boost 
low-income children’s readiness for kindergarten, by helping to prepare them for the behavioral 
expectations of school. 

This goal takes on considerable importance given that many preschool teachers report 
that they lack strategies and techniques to support children who have social-emotional challeng- 
es, and kindergarten teachers consistently rank children’s emotional and behavioral issues as a 
top concern. "‘However, the evidence base is limited on the effectiveness of interventions to im- 
prove children’s social-emotional competence, and there is even less evidence on the support 
systems that are needed to implement such interventions. To date, the evidence that does exist 
comes from small-scale tests in which the interventions’ developers are directly involved and 
which they actively oversee, providing little information about how effective the interventions 
would be when implemented on a larger scale in a wider range of settings. 

Head Start, which is the largest federally funded early childhood education program 
in the United States, aims to increase school readiness among low-income children from 
birth to age five years by boosting their cognitive, social, and emotional development. The 
Head Start CARES (“Classroom-based Approaches and Resources for Emotion and Social 
skill promotion’’) demonstration was designed to expand the current evidence base by eval- 
uating enhancements to the standard curricula that have been used in Head Start classrooms. 


'The Center on the Social and Emotional Foundations for Early Learning (CSEFEL) defines social- 
emotional development as the developing capacity of the child from birth through five years of age to 
form close and secure adult and peer relationships; experience, regulate, and express emotions in socially 
and culturally appropriate ways; and explore the environment and leam — all in the context of family, 
community, and culture. For more information, see Yates et al. (2008). 

^Dobbs, Doctoroff, Fisher, and Arnold (2006); Hawkins et al. (1999); Heckman, Stixmd, and Urzua 
(2006); Kellam et al. (1998); McClelland, Acock, and Morrison (2006); McWayne, Fantuzzo, and McDermott 
(2004); Raver et al. (2008); Reid, Eddy, Fetrow, and Stoolmiller (1999); Reynolds, Temple, and On (2010); 
Woodward and Fergusson (1999); Nagin and Tremblay (2001); Hamre and Pianta (2001). 

^Morris et al. (2013); Raver et al. (2008). 

"'La Paro and Pianta (2000). 
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The demonstration included (1) selection of three different strategies, or program “en- 
hancements,” that in smaller-scale tests showed positive effects on children’s social- 
emotional outcomes, such as reducing problem behaviors and promoting positive peer rela- 
tionships; (2) implementation of these three enhancements in many different kinds of class- 
rooms that operate within the regular Head Start system; and (3) the same professional de- 
velopment model, technical assistance, and program monitoring to support each of the three 
enhancements, in order to help ensure that they were implemented as designed while efforts 
were made to rapidly increase their scale, as Head Start CARES envisioned. 

This report, which focuses on how well the three enhancements and the related supports 
were implemented, is part of a larger Head Start CARES randomized control trial that is also 
examining the impact of the approaches on classrooms and the children in them. The Head Start 
CARES demonstration was conceived and sponsored by the Office of Head Start and the Office 
of Planning, Research and Evaluation in the Administration for Children and Families in the 
U.S. Department of Health and Human Services. The demonstration was conducted by MDRC, 
a nonprofit, nonpartisan education and social policy research organization, in collaboration with 
MEF Associates and several academic partners. 


The Head Start Cares Demonstration 

As noted above. Head Start CARES sought to provide a fair test of the effectiveness of three 
different social-emotional approaches for preschool teachers that were implemented on a large 
scale in the Head Start system. In order to conduct this test, it was important that the approaches 
be implemented with reasonable quality and intensity. While the level of support that was put in 
place to help achieve this objective was less than might be expected in small “hothouse” studies 
(which are tightly controlled and have extensive resources), the full intervention, which includ- 
ed prepared classroom materials, ongoing teacher training and coaching, technical assistance, 
and continuous monitoring, extended beyond the typical implementation of a new preschool 
intervention. ^Therefore, this demonstration approaches an “effectiveness” study, but with “effi- 
cacy” supports included: that is, it evaluates whether a fully developed intervention that has 
been shown to be efficacious under limited or ideal conditions is effective when it is imple- 
mented under more typical conditions through an independent study.® 


®For example, see Odom (2009); Wise, da Silva, Webster, and Sanson (2005). 

®See University Council for Educational Administration (n.d.) for the Institute of Education Science’s def- 
inition of an “efficacy study” versus an “effectiveness study.” 
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The Participating Head Start Grantees 

The demonstration was conducted over the course of one academic year with 17 Head 
Start grantees located across the United States/ Each grantee had at least four centers where 
classes were held, which were similar in the racial and ethnic composition of the children who 
were enrolled and in the mix of full- and part-day classrooms/ Each center generally had at 
least two classrooms that served primarily four-year-olds. Typically four or eight centers were 
grouped together as a “block,” and each center in a block was randomly assigned to one of the 
three social-emotional enhancements (the program group) or to a “business as usual” control 
group. Random assignment ensures that any observed effects, or “impacts,” on outcomes (for 
example, teachers’ classroom management skills, in this study) are a result of the intervention. 

Four grantees that participated during the 2009-2010 school year (representing 24 cen- 
ters and 78 classrooms) were in Cohort One, and 13 grantees that participated during the 2010- 
2011 school year (representing 80 centers and 229 classrooms) were in Cohort Two. A total of 
104 centers, 307 classrooms, and over 3,600 children were part of the study. This report focuses 
on the 78 centers and 230 classrooms that were assigned to receive one of the three enhance- 
ments in either Cohort One or Cohort Two. (The remaining 26 centers and 77 classrooms were 
in the control group.) 

Grantees varied on a number of characteristics, including organizational setting, geo- 
graphic location, urban/mral status, size, and racial or ethnic composition. Centers had between 
one and six participating classrooms, with an average of three classrooms participating. Average 
quality ratings in the Head Start CARES classrooms were similar to the national averages for 
Head Start on measures of classroom quality.® 

The Three Program Enhancements 

The three interventions described below are referred to as “enhancements” in this report 
because they enriched and complemented the classroom curricula and practices that already ex- 
isted in the Head Start system. The three enhancements were selected to represent three types of 
social-emotional programming. That is, while all three enhancements were aimed at children’s 


grantee is the local public or private nonprofit agency that has been designated as a Head Start agency. 

*^Two-thirds of classrooms were full day (more than 3.5 hours) and one-third were part day (3.5 hours or 
less, taught in the morning or afternoon); some teachers conducted two part-day classes per day. 

^Classroom quality was measured using the Classroom Assessment Scoring System (CLASS), a widely 
used measure of teacher-child interactions (Pianta, LaParo, and Hamre, 2008). CLASS is composed of three 
domains: Instructional Support, Emotional Support, and Classroom Organization. Head Start CARES CLASS 
scores were compared with the Head Start Family and Children Experience Survey (FACES) CLASS scores. 
For more information, see Xue et al. (2012). 
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social-emotional development, they varied in their approach to changing this set of child out- 
comes by targeting somewhat different teacher practices. 

The Incredible Years 

The Incredible Years Teacher Training Program focuses on training teachers to create 
an organized classroom climate that supports children’s ability to regulate their own behavior in 
the context of positive teacher-child relationships. The program includes problem-solving prac- 
tices, classroom organization (mles and routines), clear and consistent methods for setting limits 
with the children, a system for rewarding children’s positive behavior, praise and incentives to 
motivate students’ learning, and proactive discipline and strategies. For instance, in “circle 
time’’ (a large-group activity period), some children may be sitting quietly, ready to learn, while 
others are playing with their friends and yelling. An Incredible Years teacher might say, “I real- 
ly like the way Juan is sitting with his hands in his lap,’’ instead of mentioning the children who 
are misbehaving. 

Preschool PATHS 

Preschool PATHS (Promoting Alternative Thinking Strategies) focuses on training 
teachers to use clearly outlined lessons to improve children’s social problem-solving skills, in- 
cluding the ability to recognize and regulate emotions, define problems, and engage in “antici- 
patory planning,’’ which considers the consequences of various reactions to problems. Teachers 
can use PATHS’ weekly lesson plans during circle time, and they can incorporate other activi- 
ties throughout the rest of the day, such as singing a song about an emotion or painting different 
emotions on faces, to give children opportunities to practice the targeted skills. In a PATHS 
classroom, teachers talk about their feelings and encourage children to think about their and 
others’ feelings. For example, if two children are playing nicely together, the teacher might talk 
about the emotions they had been discussing in their lesson that day. She might say, “How do 
you think Ann felt when Neveah gave her a hug? That’s right, she felt happy! How do you look 
when you feel happy? What makes you feel happy?’’ 

Tools of the Mind — Play 

Tools of the Mind focuses on training teachers to use adult-supported “make-believe’’ 
play and other activities to strengthen children’s ability to regulate their own behavior, emo- 
tions, and thoughts (“self-regulation’’).'® The Tools program requires the teacher to restmcture 
the room and the school day, devoting large blocks of time to planning and enacting role- 


*®Make-believe (or “pretend”) play is a form of high-level play in which children use their imaginations to 
role-play, become different “characters,” play out different stories, and enact various scenarios that rely on and 
encourage creativity. 
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playing games. A central component of Tools, for instance, is a daily 50-minute period of 
make-believe play to enhance children’s planning skills, understanding of social roles (such as 
“parent,” “friend,” or “family”), memory and focused attention, and social-emotional under- 
standing. This time is characterized by “scaffolding,” in which the teacher supports and encour- 
ages the children in their attempts to take on a challenging task or acquire a skill that is just be- 
yond their current level of ability. For example, a child might first draw a picture showing that 
she intends to play house and will be the mother. The teacher would then help the child write 
out and expand on her plans, asking, “What will you do as the mother? How could you make 
dinner for your children?” She would help the child come up with a more complex role-play 
scenario. Then, while the child is playing, the teacher might help her expand the role-play even 
further, asking questions such as, “What might you need before you are able to cook dinner? 
How would you get to the grocery store?” In this way, the teacher helps the child to build self- 
regulation, including mental flexibility, memory, and inhibition of automatic responses, by cre- 
ating a plan, sticking with a role for an extended period of time, and shifting between her own 
perspective and the perspective of the character she is pretending to be. 

The Professional Development Model 

The professional development model for the Head Start CARES demonstration includ- 
ed structured teacher training with follow-up coaching in the Head Start classrooms. 

Teacher training included the use of well-developed manuals and other materials for 
each of the three enhancements; delivery of four to six training sessions to both lead and assis- 
tant teachers and to coaches throughout the school year; and trainers’ support of coaches and 
teachers through classroom visits two to three times a year. 

Coaching involved 60 minutes of scheduled in-classroom observation and a 30-minute 
meeting each week with the lead and assistant teachers jointly. Coaches, who were selected by 
the grantees, reflected with teachers on their practice, identified and helped resolve issues that 
hindered implementation of the enhancements, demonstrated techniques to use in the class- 
room, and set goals and planned for the next week. Fifty-two coaches provided support to an 
average of four classrooms each. The Head Start CARES coaching component was designed to 
focus on the teachers’ practices, foster a collaboration between the coach and both teachers, 
provide insfruction to teachers, identify areas for support, and place the coach in an evaluative 
but nonsupervisory position. 

Technical Assistance and Program Monitoring 

MDRC provided technical assistance to the developers of the three enhancements, 
trainers, coaches, teachers, and a member of each grantee’s staff (called the “grantee liaison”). 
The technical assistance included the development of materials to support grantees’ selection of 
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coaches, a toolkit to support the coaching process, a launch meeting to introduce all key stake- 
holders to the demonstration and the enhancements, support for planning the teacher training 
sessions, using data from a management information system (MIS) to determine what kind of 
technical assistance was needed, regular conference calls for coaches across the grantees, 
MDRC site teams’ regular check-ins with grantee liaisons, and MDRC technical assistance 
team’s check-ins with enhancement developers. 

Program monitoring, also provided by MDRC, included use of the MIS, which served 
as a repository for coaches and trainers to submit information about how teachers were imple- 
menting the enhancements and how often coaches, teachers, and trainers interacted. The system 
included online weekly and monthly surveys. As part of ongoing monitoring, classrooms were 
rated on a scale of 1 (low) to 5 (high), with a rating at or above a 3 considered to be satisfactory. 


Findings 

This report focuses on whether the professional development supports and the in-classroom 
social-emotional enhancements were implemented at scale with fidelity to the original design — 
that is, in the amount and with the quality that the developers intended, as described in Box ES. 1 . 

Overall, high levels of participation in the training and coaching components supported 
satisfactory levels of classroom implementation of all three enhancements, which received fidel- 
ity scores of 3 or higher. Despite challenges and some variation in the fidelity with which the 
three enhancements were implemented in the classroom, each enhancement led to the changes 
in teacher practices that it was designed to influence. Thus, it appears that the demonstration did 
ensure a fair test of large-scale implementation of the three enhancements, thereby providing a 
sound basis for evaluating their impact on children and classrooms in the Head Start system. 

Fidelity to the Professional Development Model for Teacher Training and 

Coaching 

On the whole, teacher training and coaching were delivered as intended to support im- 
plementation of the enhancements in the classroom. 

• Training quality and attendance were generally strong across all grant- 
ees and enhancements. 

Teacher attendance at the training sessions was high, with 93.5 percent of classrooms 
sending a lead teacher to training, and lead and assistant teachers attending together 82.4 percent 
of the time. Teachers reported that the training material was presented in an accessible way and 
that the trainers were supportive and concerned about their progress. Teachers also reported that 
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Box ES.1 

Defining Fidelity in Head Start CARES 


Fidelity in Head Start CARES focused on two components: implementation fidelity, mean- 
ing fidelity to the professional development model; and intervention fidelity, meaning fideli- 
ty to the design of the enhancement. 

Implementation Fidelity: Fidelity to the Professional Development Model 

Training Fidelity, Fidelity to the training component was achieved when grantees offered 
all of the planned training sessions, trainers delivered the full content of the training, and lead 
and assistant teachers attended enhancement training sessions together. Fidelity was assessed 
by examining (1) the dosage (that is, frequency and duration) of training received; (2) the 
quality of the training received; and (3) the trainer support received. 

Coaching Fidelity, Fidelity to the coaching component was assessed by examining (1) the 
dosage of coaching received in classrooms; and (2) the quality of the coaching received in 
classrooms. 

Intervention Fidelity: Fidelity of Classroom Implementation 

Fidelity to the enhancement design was rated on a scale of 1 (low) to 5 (high), with a rating 
of 3 considered satisfactory. The rating reflected whether the teachers, children, and class- 
rooms were clearly steeped in an evidence-based, social-emotional enhancement. For exam- 
ple, one item in the scale was, “The children are actively engaged in [specific enhancement] 
throughout the day. It is not just seen as a special event.” Fidelity to the enhancement as de- 
signed was measured in ter ms of dosage and quality: high fidelity indicated that the en- 
hancement was implemented exceptionally well and often, and low fidelity indicated that the 
enhancement was implemented poorly or rarely. 


the training supported their relationships with the coaches and prepared them to successfully 
implement the enhancement’s practices with their students. 

• Coaching dosage (frequency and duration) and coaching quality were 
generally high. 

Coaches met with teachers weekly to reflect on implementation for 5 1 minutes on aver- 
age, which was longer than expected. Coaches met with their trainers for feedback and support 
as expected, about two to three times a month on average, although this rate varied by each en- 
hancement’s plan for coach support. Trainers found the coaches to be of moderately high quali- 
ty — rating them 3.97 out of a possible 5, meaning that the coaches were professional, knowl- 
edgeable, and attentive to teachers’ needs. Incredible Years and Tools of the Mind trainers gave 
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moderately high ratings to their coaches (3.74 and 3.89, respectively), and Preschool PATHS 
trainers gave a high rating to their coaches on average (4.30). 

Fidelity to the Enhancement Design 

• All three enhancements were generally well received by teachers and 
other staff. 

Teachers reported that the enhancements made sense to them and that they were able to 
implement them in the classroom as the developers intended. Teachers did, however, feel that 
some of the less scripted, more theoretical enhancement components (like supporting make- 
believe play) were more difficult to implement than the more highly scripted activities (such as 
using written lessons). When asked in interviews how children responded to the enhancements, 
lead and assistant teachers, center directors, and other center-based staff believed that children 
benefited from them. 

• Coaches and trainers reported that each of the enhancements was im- 
plemented with satisfactory fidelity, as defined by quality and dosage. 
Fidelity improved over time. 

Both coaches and trainers evaluated fidelity of classroom implementation over the 
course of the year, with the average Head Start CARES classroom scoring 3.47 out of a possible 
5. As rated by coaches, fidelity improved between September and April from acceptable im- 
plementation (around a score of 3) to proficient implementation (around a score of 4). Coaches 
rated fidelity somewhat higher than trainers did. Coaches reported that most (83 percent) Head 
Start CARES classrooms scored higher than the “satisfactory” threshold of 3 in January, and in 
April, 60 percent of Head Start CARES classrooms had scored higher than 4, indicating that 
they were implementing the enhancements well and consistently. 

• Fidelity in the classroom varied somewhat across the three interven- 
tions, The structure, goals, and activities that are intrinsic to each en- 
hancement shaped their implementation. 

As rated by coaches and trainers, the average fidelity score for The Incredible Years 
(3.69 over the year) and Preschool PATHS (3.73) exceeded the “satisfactory” threshold of 3. 
Tools of the Mind implementation was not as strong (an average of 2.97 over the year), but was 
close to the rating of 3 that was considered satisfactory. Additionally, data about the ease and 
challenges of implementing the enhancements were synthesized from site visits, group technical 
assistance calls with coaches, and interviews with teachers, coaches, and trainers. Themes 
emerged from this review that reflected how the structure, goals, and activities of each en- 
hancement were integral to the way in which teachers implemented it. 
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The Incredible Years 

The Incredible Years enhancement may have been relatively easy for teachers to im- 
plement because they did not need to alter their classroom schedules to allow for additional les- 
sons and activities. However, it may have been more difficult for teachers who had been using 
specific classroom management techniques to integrate the Incredible Years techniques into 
their practice. Incredible Years implementation required teachers to be mindful of their mo- 
ment-to-moment interactions with children, and in some cases they had to break old habits and 
develop new ones. 

Preschool PATHS 

Preschool PATHS was also relatively easy for teachers to implement because it was 
generally highly structured, took up a defined and limited amount of time each week, and was 
well scripted. PATHS included a regular daily activity, a weekly lesson during circle time, a 
weekly activity that extended the concept from the week’s lesson into other parts of the day, and 
generalization, in which teachers integrate the themes and core practices of the specific lesson 
into different activities throughout the day and week. 

Tools of the Mind 

Tools of the Mind required teachers to physically rearrange their classrooms and im- 
plement a block of Tools-specific activities daily. While some of the activities had helpful man- 
uals and supportive materials, and were easy to implement, other activities required a consider- 
able amount of effort by the teachers, including their acceptance of the enhancement’s general 
philosophy. Tools also required a high degree of teacher focus and individualized attention for 
each child; instead of doing paperwork during the children’s play time, for example, which is 
fairly typical, the teachers had to actively engage children in playing make-believe and had to 
intentionally scaffold their efforts. 

Implementation Challenges 

• While teachers found it relatively easy to implement The Incredible 
Years and Preschool PATHS, they still faced some significant implemen- 
tation challenges across all three enhancements. 

Challenges implementing the enhancements included a lack of basic classroom supplies 
(such as books or play clothes), as well as the need for translation of enhancement materials and 
manuals into Spanish. Teachers who taught two part-day sessions daily had difficulty finding 
time to prepare and meet with coaches during the school day, although they had more flexibility 
at the end of the week because they generally taught Monday through Thursday only. Addition- 
ally, teachers had only limited time to talk and share materials and support across classrooms. 
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and there was little collaboration within centers, which made it difficult for teachers to work 
together to find solutions to the implementation challenges they experienced. Organizational 
requirements that were triggered by, for example, new curricula or assessments made it difficult 
for teachers to make time to implement Head Start CARES. In addition. Head Start programs 
must comply with Head Start Program Performance Standards and be monitored regularly, 
which may have contributed to teachers’ stress or feeling that study participation was just one 
more burdensome task. 

• The level of organizational support that grantees provided for the en- 
hancements influenced fidelity to the original design when the enhance- 
ments were delivered in the classroom. 

The Head Start CARES data do not permit a formal test of the relationship between or- 
ganizational capacity and overall fidelity. However, organizational support in the demonstration 
was evaluated from many different perspectives. In order to identify characteristics of a strong 
grantee, assessments by coaches, trainers, developers, and the research team were combined into 
a single rating system. These ratings suggest that organizational capacity played a key role in 
ensuring quality professional development and classroom implementation of the enhancements. 

The grantees that demonstrated greater fidelity sent teachers strong messages of support 
for the enhancements, hired coaches with appropriate qualifications in a timely manner, provid- 
ed administrative supervision and support for coaches, supported efforts to cross-walk and inte- 
grate the enhancement with their core curriculum, made time and space available for teacher 
training, and devised creative solutions to time management challenges. Among grantees that 
were rated lower on organizational support, messaging and communication between center and 
grantee staff were consistently cited as a challenge. The research team’s perception was that 
grantee “buy-in” of and opinions about specific enhancements also seemed to influence imple- 
mentation in the classroom. 

Did Implementation of the Enhancements Lead to Changes in Teacher 
Practice? 

• Each enhancement emphasized a specific teacher practice. As hypothe- 
sized, these practices improved in the program group classrooms, where 
the enhancements were implemented, compared with the control group 
classroom s . 

In the Head Start CARES demonstration, changes in teacher practice were conceptual- 
ized as “first order” effects; that is, teacher practice was considered the direct target of these en- 
hancements. It was hypothesized that in order to see change in “second order” outcomes such as 
classroom interactions or children’s social-emotional behaviors, change would first need to be 
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observed in the “first order” practices. Information about teacher practice was collected through 
an independent classroom observation called the Adapted Teaching Style Rating Scale (Adapted 
TSRS)." The measure focused on three distinct practices, representing each of the enhance- 
ments, and rated classrooms from 1 (low) to 5 (high).'^ The hypothesis for each enhancement is 
outlined below, and the results of the Adapted TSRS observations appear in Table ES. 1 . 

The Incredible Years 

The Incredible Years enhancement was expected to affect teachers’ classroom man- 
agement, which includes the ability to apply consistency and routines, preparedness for the day, 
awareness of what is happening around the classroom, and the use of positive behavior man- 
agement practices instead of negative behavior management practices — that is, focusing on 
positive behavior (“Juan is sitting quietly”) or using incentives to encourage good behavior ra- 
ther than reprimanding children for negative behavior (“Timmy, stop yelling!”). In fact, on av- 
erage, The Incredible Years resulted in small to moderate improvements in classroom manage- 
ment scores and social-emotional instruction scores in the centers that were in the program 
group, compared with the control group centers’ average scores. These impacts are approxi- 
mately the same magnitude as the lower range of effects that were observed in two smaller, 
more intensive trials of The Incredible Years (CSRP, formerly known as the Chicago School 
Readiness Project, and the Foundations of Learning demonstration).'^ 

Preschool PATHS 

PATHS was expected to affect teachers’ social-emotional instruction, which includes 
modeling appropriate labeling (that is, identification) and regulation of emotions, supporting 
children’s expression of emotions (whether positive or negative), encouraging children to regu- 
late their emotions, facilitating children’s social awareness and empathy, and teaching children 
social problem-solving skills. In fact, PATHS had a large positive impact on social-emotional 
instruction scores in program group centers compared with the control group centers’ average 
scores. This impact was comparable to the upper range of effects found in a previous trial of 
PATHS and a language intervention (REsearch-based, Developmentally Informed, or REDI). 


"The Adapted TSRS was created by C. Cybele Raver, Celene E. Domitrovich, Mark T. Greenberg, 
Pamela A. Morris, and Shira Kobiik Mattera as part of the Head Start CARES demonstration. 

*^This scale is different from the one described earlier, on page 6, which measured fidelity. 

*^See Raver et al. (2008) for the CSRP study, formerly known as the Chicago School Readiness Project 
(not associated with The Chicago School®, which is a trademark of The Chicago School of Professional Psy- 
chology), and Morris et al. (2013) for the Foundations of Learning study. 

'“'Domitrovich et al. (2009). 
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Head Start CARES Demonstration 


Table ES.l 


Classroom-Level Impacts: Teacher Practice Observations 




Incredible Years (lY) 

Preschool PATHS 


Tools of the Mind 



Control 

Program Difference 


Program 

Difference 


Program 

Difference 



Group 

Group 

(lY vs. 

Effect 

Group 

(PATHS vs. 

Effect 

Group 

(Tools vs. 

Effect 

Component 

Rating 

Rating 

Control) 

Size" 

Rating 

Control) 

Size" 

Rating 

Control) 

Size" 

Classroom management 

3.79 

4.09 

0.30 ** 

0.44 

3.90 

0.12 

0.17 

3.89 

0.10 

0.15 

Social-emotional instruction 

1.76 

1.98 

0.22 * 

0.30 

2.42 

0.66 *** 

0.92 

1.78 

0.02 

0.02 

Scaffolding 

1.44 

1.41 

-0.03 

-0.06 

1.48 

0.05 

0.09 

1.78 

0.35 *** 

0.68 

Sample size'’ 











Center 

26 

26 



26 



26 



Classroom 

77 

77 



77 



76 




SOURCE: MDRC calculations based on observational assessments completed using the Adapted Teaching Style Rating Scale (Adapted TSRS). 

NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

All models are based on pooled analyses of program group status, controlling for the pretest and blocking variable used to randomly assign 4, 8, 
or 12 centers to the program group. 

Each component was rated on a scale of 1 (low) to 5 (high), reflecting fldelity of classroom implementation. 

“Effect size is calculated by dividing the impact of the program (the difference between the means for the program group and the control group) by 
the standard deviation for the control group. 

'’For all variables in the table, data are available for 100 percent of the sample. 



Tools of the Mind 


Tools of the Mind was expected to affect teachers’ scaffolding of peer interactions and 
play by helping children plan for and expand their activities in ways that are challenging or 
novel. In fact, Tools exerted a moderate to large increase in the average scaffolding scores in the 
program group centers, compared with the control group centers’ average scores. In a previous 
small-scale Tools trial, effect sizes for the early childhood environment and teachers’ scaffold- 
ing practices were substantially larger than those found in this study. 

Summary and Conclusion 

The Head Start CARES enhancements were implemented at scale as intended — that is, with 
fidelity to their design — and led to the hypothesized effects on teacher practice. Notably, a 
substantial support system helped to achieve these results. The support system included inter- 
ventions that had prepared manuals and other materials to use in the classroom, knowledgeable 
trainers, and intensive coaching, as well as technical assistance and ongoing monitoring by a 
centralized entity using a flexible MIS. Fidelity was also facilitated when all involved parties 
demonstrated a sustained and coordinated commitment, including a clear and consistent mes- 
sage from the grantees’ administration about the importance of the intervention. 

In Head Start CARES, the theory of change specifies how the enhancements are hy- 
pothesized to change child outcomes at the end of the preschool year, as follows: (1) implemen- 
tation of the enhancements with fidelity would strengthen existing teacher practices and/or lead 
to changes in teacher practice; (2) changes in teacher practice would lead to improved class- 
room interactions; and (3) improved classroom interactions would lead to improved child out- 
comes. This report focuses on the first of those three steps, and concludes that each enhance- 
ment was implemented with fidelity to its developer’s design, which led to the expected chang- 
es in teacher practice. A separate report on the Head Start CARES demonstration examines the 
impacts of the three enhancements on teacher-child interactions and children’s outcomes.'*’ 


'^Barnett et al. (2008). 

'^Morris, Mattera, Castells, and Bangser (forthcoming). 
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Chapter 1 

Introduction 


Social-emotional skills and competencies are cmcial to young children’s development. The 
Center on the Social and Emotional Foundations for Early Eeaming defines social-emotional 
development as the developing capacity of the child from birth through five years of age to form 
close and secure relationships with adults and peers; experience, regulate, and express emotions 
in socially and culturally appropriate ways; and explore the environment and learn — all in the 
context of family, community, and culture. ' 

Unfortunately, low-income children face a number of risks to their social-emotional de- 
velopment, generating gaps in social-emotional functioning between these children and their 
more affluent peers that are observed at the start of children’s formal schooling and remain or 
increase during the elementary school years. ^ One potential pathway to closing this gap and 
boosting children’s readiness for school is to intervene directly by preparing them to receive 
insttuction and take better advantage of learning opportunities,^ and to get along better with 
teachers and peers.'* 

Kindergarten teachers rank children’s social-emotional competence and its associated 
behavioral challenges as their top concern.^ Yet, many preschool teachers report that they lack 
skills to address the behavioral and emotional challenges that they encounter in their class- 
rooms.*’ Only a small base of evidence exists about specific interventions to bolster children’s 
social-emotional competencies, and even less evidence exists for the various support systems 
that have been (or could be) put in place to implement those interventions.^ Such evidence 
comes from small-scale evaluations in which the developers of the interventions were directly 
involved and that they actively oversaw, yielding little information about their potential effec- 
tiveness on a national scale and in a larger and more diverse set of classrooms. Additionally, 
these interventions each used unique support systems to facilitate their implementation, again 
providing little guidance about how to consistently support implementation of social-emotional 
programs on a large scale. 


'Yates et al. (2008). 

^Alexander, Entwisle, and Kabbani (2001); Brooks-Guim, Duncan, and Aber (1997); Dodge, Pettit, and 
Bates (1994); Entwisle and Hayduk (1988); Harare and Pianta (2001). 

^McLelland, Morrison, and Holmes (2000); Raver, Gamer, and Smith-Donald (2007). 

''Morris et al. (2010); Raver et al. (2008). 

^La Paro and Pianta (2000). 

*’Lloyd and Bangser (2009). 

^Bierman et al. (2008a); Consortium on the School-Based Promotion of Social Competence (1994); Raver 
et al. (2008); Webster- Stratton, Reid, and Hammond (2001). 
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The Head Start CARES (“Classroom-based Approaches and Resources for Emotion 
and Social skill promotion”) demonstration was designed to strengthen the evidence base for 
programs that focus on young children’s social-emotional competence. The demonstration im- 
plemented and tested three promising interventions that were designed to support children’s 
social-emotional competencies in a large number of diverse classrooms that operated within the 
Head Start system, a federal program that aims to increase school readiness among low-income 
children from birth to age five years by boosting their cognitive, social, and emotional devel- 
opment. The same comprehensive model of professional development, technical assistance, and 
program monitoring was used to support each of the three approaches, in order to help ensure 
that they were implemented as designed. The support system was especially important because 
of the rapid expansion, or “scale-up,” that the demonstration required for full implementation, 
which involved 104 Head Start centers with 307 classrooms in total. 

This report, focusing on how well the three interventions and the support system were 
implemented, is part of a larger study of the Head Start CARES demonstration that also exam- 
ines the impact of the approaches on classrooms and the children in them. The demonstration 
was conceived and sponsored by the Office of Head Start and the Office of Planning, Research 
and Evaluation in the Administration for Children and Families in the U.S. Department of 
Health and Human Services. The demonstration was conducted by MDRC, a nonprofit, nonpar- 
tisan education and social policy research organization, in collaboration with MEF Associates, 
Survey Research Management, and several academic partners. 

Most notably, the findings demonstrate that it is possible to implement well-stmctured 
social-emotional interventions on a large scale in a range of Head Start classrooms across the 
country. A separate report on Head Start CARES presents the impacts of each of the interven- 
tions on teacher-child interactions and on outcomes for children.* 


The Head Start CARES Demonstration 

Head Start CARES built on compelling research that shows that a focus on social-emotional 
competence is critical for preschool quality as well as children’s longer-term development. As 
noted above, a primary goal of the demonstration was to ensure a fair test of the large-scale im- 
plementation of three different social-emotional interventions. The demonstration included a 
systematic framework to support implementation scale-up and replication, including program 
packages consisting of prepared manuals and other materials to use in the classroom, ongoing 
teacher training and coaching, technical assistance, and performance feedback. 


*Morris et al. (forthcoming). 

’Dobbs, Doctoroff, Fisher, and Arnold (2006); Hawkins et al. (1999); Kellam et al. (1998); McWayne, 
Fantuzzo, and McDermott (2004); Raver et al. (2008); Reid, Eddy, Fetrow, and Stoolmiller (1999). 
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The three evidence-based, social-emotional interventions selected for the Head Start 
CARES demonstration are The Incredible Years Teacher Training Program,'” Preschool 
PATHS (Promoting Alternative Thinking Strategies)," and Tools of the Mind — Play.'^ The 
demonstration referred to the three interventions as “enhancements” because they were intended 
to enrich and complement practices and curricula that already existed in Head Start classrooms. 

The three enhancements were selected to represent three types of social-emotional pro- 
gramming. That is, while all three enhancements were aimed at children’s social-emotional de- 
velopment, they varied in their approach to changing this set of child outcomes by targeting 
somewhat different teacher practices. Enriching social-emotional development is implicit in the 
Incredible Years classroom management and behavior management approach; Preschool 
PATHS uses an instmctional approach to improving social-emotional development through 
stmctured lessons that are designed to teach children ways to think about emotions and respond 
to peers in social interactions; and Tools of the Mind promotes children’s learning and regula- 
tion of behaviors, emotions, and thoughts through stmctured “make-believe” (or “pretend”) play 
activities — a form of high-level play in which children use their imaginations to role-play, be- 
come different “characters,” play out different stories, and enact various scenarios that rely on 
and encourage creativity. 


Testing Social-Emotional Enhancements in the Context of Head 
Start 

Head Start, the largest federally funded early childhood program in the United States, is particu- 
larly well suited for testing varied approaches to improving teachers’ classroom practices. The 
program addresses the needs of low-income families and children in order to narrow the gap 
between disadvantaged children and their higher-income peers. Head Start provides early child- 
hood education and care combined with comprehensive services during the preschool period to 
improve children’s social competence, health, and academic readiness for school. 

Because of its origin as a community-based program. Head Start provides an especially 
useful setting for testing the scale-up of evidence-based enhancements in a range of contexts. 
While focused on the common goal of serving low-income children and families using a “whole 
child” approach,'^ Head Start programs reflect a wide range of quality, resources, and pedagog- 

'”The Teacher Classroom Management Program, one of three Incredible Years programs, was studied in 
Head Start CARES. 

"Domitrovich, Greenberg, Kusche, and Cortes (2004). 

'^Barnett et al. (2008); Domitrovich, Cortes, and Greenberg (2007); Webster-Stratton, Reid, and Ham- 
mond (2004). 

'^A “whole child approach” focuses on all aspects of development — physical, social-emotional, and 
cognitive. 
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ical theories that could interact with new classroom practices to support or hinder implementa- 
tion. Moreover, Head Start programs are offered in mral, suburban, and urban contexts, serving 
a diverse group of families across the country. Evaluating the implementation of evidence-based 
enhancements in these varied contexts is important for determining whether they can be deliv- 
ered well, if they can universally support children’s social-emotional development, and under 
which contexts they can be implemented most effectively. 


Characteristics of Participating Grantees 

The Head Start CARES demonstration took place in 104 Head Start centers (each of which 
houses a varying number of classrooms) with 307 classrooms in total, administered by 17 
grantees/delegate agencies (referred to as “grantees” throughout this report) — the local public 
or private nonprofit agencies that received federal funding to operate Head Start programs — 
located across the country. The three enhancements were implemented within the context of a 
randomized control trial in order to obtain rigorous evidence of their effectiveness. The demon- 
stration, which operated from 2009 to 2011, used a lottery-like process to randomly assign the 
Head Start centers to one of the three different social-emotional enhancements or to a “business 
as usual” control group. (See Figure 1.1.) Random assignment ensures that any differences (or 
“impacts”) that are observed in outcomes between the program and control groups can be at- 
tributed with a high level of confidence to the intervention that is being evaluated. For each 
grantee, at least four centers were randomly assigned to one of the three different social- 
emotional enhancements or to a control group; all three enhancements were tested by each 
grantee, but no more than one enhancement was implemented in each center. In each case, 
teachers were trained in the enhancement that was assigned to their center and then were pro- 
vided with coaches, who helped them implement the new practices in their classrooms. The in- 
tervention was conducted for one year by each grantee. 

As discussed in more detail in Appendix A, the selection of grantees was a multi-step 
process that began with a population of potentially eligible Head Start grantees identified from a 
Program Information Report (PIR) database, which is held by the Office of Head Start. Grant- 
ees were stratified by region of the country, raciaPethnic composition of child enrollment, and 
whether they were located in a metropolitan or mral area, or spread across both metropolitan 
and nonmetropolitan environments (a measure known as “urbanicity”'^). Region was defined as 
one of four groups: Northeast, South, Midwest/Plains, and West. Four grantees in the Northeast 


'"'See Appendix A for more information on racial/ethnic designations. 
'^See Appendix A for more information on urbanicity. 
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Head Start CARES Demonstration 


Figure 1.1 

Randomization Design 



NOTE: “Nine grantees had 4 participating centers each; seven grantees had 8 participating centers each; 
and one grantee had 12 participating centers. 


participated in the Head Start CARES demonstration during the 2009-2010 school year, form- 
ing Cohort One, and 13 grantees from the rest of the country participated during the 2010-201 1 
school year, forming Cohort Two. As shown in Figure 1.2, the 17 Head Start grantees that were 
selected to participate in the Head Start CARES demonstration were located in 10 states across 
the nation and were distributed fairly equally across the four regions of the country, with four 
grantees in the Northeast, four in the West, three in the South, and six in the Midwest/Plains. 

The grantees varied considerably in terms of their venue, size, and neighborhood con- 
text. More than one-third of grantees were situated within community action agencies,'*’ one- 
third were operated by stand-alone nonprofit entities, and about one- fourth were located in large 
local school systems. The grantees were spread approximately evenly among three size catego- 
ries: about one-third of grantees were “small” (enrolling fewer than 800 children), one-third 


*^A community action agency is either a public or private nonprofit organization, funded primarily by a 
Community Services Block Grant (CSBG) to administer and coordinate programs on a communitywide basis. 
These agencies provide services that address the full range of family needs, from child development programs, 
to youth and adult employment and training programs, and services for seniors. Stand-alone nonprofit entities 
are organizations without governmental affiliations, such as religious institutions or nonprofit hospitals. For 
more information, see National Community Action Foundation (2011). 
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were “medium” (enrolling between 800 and 1,500 children), and another third were “large” (en- 
rolling more than 1,500 children). Across the two cohorts, nine grantees had 4 participating cen- 
ters each, seven grantees had 8 participating centers each, and one grantee had 12 participating 
centers. 


The centers were located in a variety of neighborhoods. Just under half (48 percent) of 
the centers were in primarily residential areas. Observers, who did not know the intervention 
status of the classrooms, were asked to rate the neighborhood context during site visits; these 
observers indicated that the centers were generally situated in safe areas and were in fair condi- 
tion — that is, they mostly felt comfortable in the area, and the buildings and homes that sur- 
rounded the center of the neighborhood were in slightly better than fair condition. 


Characteristics of Participating Centers 

Between one and six classrooms in each Head Start center participated in the Head Start 
CARES demonstration, with an average of three classrooms participating per center. While the 
grantees varied, a typical Head Start CARES classroom had a minimum of one lead teacher and 
one assistant teacher. Two-thirds of classrooms were full day (more than 3.5 hours), and one- 
third of classrooms were part day (3.5 hours or less, in the morning or afternoon). In compari- 
son, 54 percent of the children in programs that are based in Head Start centers nationally are 
enrolled in full-day programs, and 46 percent are in part-day programs. Some of the part-day 
classrooms in the demonstration operated as double sessions — that is, teachers taught one class 
in the morning and a second, separate class to another group of children in the afternoon. In 
double-session classrooms, only the first session was included in the research study. 

The Head Start CARES social-emotional enhancements were implemented on top of 
“base curricula” that varied by grantee. As is typical in Head Start, where nearly 70 percent of 
programs use the Creative Curriculum or High/Scope curricula,'^ the majority of Head Start 
CARES grantees used Creative Curriculum (12) or High/Scope (3). Of the remaining two 


'Wikens et al. (2011). The Creative Curriculum for Preschool is based on the following five fundamental 
principles, which guide practice and help intentionally set up preschool programs: (1) positive interactions and 
relationships with adults provide a critical foundation for successful learning; (2) social-emotional competence 
is a significant factor in school success; (3) constructive, purposeful play supports essential learning; (4) the 
physical environment affects the type and quality of learning interactions; and (5) teacher-family partnerships 
promote development and learning. See www.creativecurriculum.net for more information. High/Scope is a 
comprehensive curriculum and teaching practice that focuses on six dimensions of school readiness: (1) ap- 
proaches to learning; (2) language, literacy, and communication; (3) social and emotional development; (4) 
physical development; (5) health and well-being; and (6) arts and sciences. Children engage in both individual 
and social play, participate in small and large groups, assist with clean-up, socialize during meals, develop self- 
care skills, and exercise their small and large muscles. See www.highscope.com for more information. 
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grantees, one reported that High/Scope and a state-adapted core curriculum served as the prima- 
ry curricula, and the other grantee reported that DLM Early Childhood Express was the primary 
curriculum.'* 

Head Start CARES teachers were relatively typical of the general population of Head 
Start teachers. As shown in Table 1.1, lead teachers located in Head Start CARES classrooms 
were predominantly female (96 percent), had at least a bachelor’s degree (62 percent), and were, 
on average, 43 years of age. The majority (63 percent) had taught for at least 10 years. Over half 
(64 percent) of Head Start lead teachers nationally have at least a bachelor’s degree, and the av- 
erage Head Start teacher has been in the classroom for nearly nine years (not shown in table). 

As shown in Table 1.2, Head Start CARES classrooms were also similar to classrooms 
in a nationally representative study of Head Start centers on the three most widely used dimen- 
sions of the Classroom Assessment Scoring System (CEASS) — Preschool Version,^" a nation- 
ally utilized measure of classroom quality that focuses on three domains: Instmctional Support, 
Emotional Support, and Classroom Organization. Nationally, scores on the CEASS Instmction- 
al Support domain are usually the lowest, with more elevated Emotional Support and Class- 
room Organization scores indicating that preschool classrooms are generally warmer emotional- 
ly and more highly stmctured than they are academically focused.^' 

Head Start serves a racially and ethnically diverse group of children, with about a third 
nationally identified as African-American and a third as Hispanic. The Head Start CARES 
sample was similarly diverse, with 33 percent of the children identified as non-Hispanic, Afri- 
can-American, and 43 percent identified as Hispanic. Hispanic children were slightly overrepre- 
sented, while non-Hispanic, white/other children were underrepresented (32 percent nationally 
versus 24 percent in the study sample). 


'*The DLM Early Childhood Express offers a comprehensive, child-centered curriculum with strong 
teacher support consisting of daily “read-alouds” to enrich students’ imagination; nonfiction focus to build 
background, vocabulary, and oral language; rhymes, songs, and dances to develop phonological awareness; 
“how-to science” to teach observing and investigating; manipulatives (hands-on objects) and games to convey 
math and science concepts; and social emotional instruction to develop interpersonal skills. See 
www.mheonline.com for more information. 

'^U.S. Department of Health and Human Services, Administration of Children and Families (2012). 

^"See West et al. (2011) for the nationally representative study of Head Start centers. See La Paro, Pianta, 
and Stuhlman (2004) for a discussion of CLASS. 

^’National Center on Quality Teaching and Learning (n.d.); Minnesota Early Learning Foundation, Re- 
search Consortium (2011). 
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Table 1.1 


Selected Baseline Characteristics for Overall Lead Teacher Sample 


Characteristic 

Full Sample 

Age (years) 

42.87 

Female (%) 

96.17 

Race and ethnicity (%) 

White, non-Hispanic 

27.21 

African-American, non-Hispanic 

37.81 

Hispanic 

29.33 

Other/multiraciaF 

5.65 

Education (%) 

Lower than an associate’s degree 

11.34 

Associate’s degree but no bachelor’s degree 

27.15 

Bachelor’s degree but no graduate degree 

53.61 

Graduate degree 

7.90 

Teaching experience (%) 

Less than 3 years 

6.97 

3 years to <10 years 

30.31 

At least 10 years 

62.72 

Sample size*’ 

307 


SOURCE: MDRC calculations based on the spring lead teacher self-survey (baseline). 

NOTES: “““Other” includes Asian, Native Hawaiian/Pacific Islander, and American 
Indian/ Alaska Native. 

’’For all variables in the table, data are available for at least 90 percent of the sample. 


Ensuring a Fair Test of the Enhancements in the Context of 
Scaie-Up 

Before the Head Start CARES demonstration was launched, each of the three enhancements 
had been shown to have positive impacts in at least one randomized control trial with low- 
income populations that were similar to those in Head Start. Four randomized control trials of 
the Incredible Years Teacher Training Program have been conducted in the United States with 
low-income children and published, two of which combined the Incredible Years teacher pro- 
gram with clinical classroom consultation and stress management training. Impacts on chil- 
dren’s social-emotional outcomes in the latter two trials ranged from 0.22 to 0.43 of a standard 
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Head Start CARES Demonstration 


Table 1.2 


Classroom Climate of Head Start CARES and National Samples 


Domain 

CLASS Score" 

Head Start CARES FACES 2009’’ 

CLASS Instructional Support 

2.5 

2.3 

CLASS Emotional Support 

5.2 

5.3 

CLASS Classroom Organization 

4.7 

4.7 

Sample size (classrooms)'^ 

307 

370 


SOURCE: MDRC calculations based on observational assessments completed using the 
Classroom Assessment Scoring System (CLASS); see Pianta, LaParo, and Hamre (2008). 

NOTES: “The CLASS score is based on a scale of 1 (low) to 7 (high). 

'’The Head Start Family and Child Experiences Survey (FACES) is a nationally 
representative study of program performance (Moiduddin et al., 2012). 

‘’For all variables in the table, data are available for at least 98 percent of the Head Start 
sample. 


deviation — in other words, the program had effects ranging from small to moderate on a num- 
ber of social-emotional outcomes. Preschool PATHS alone or with a literacy or professional 
development component has been tested in three randomized control trials involving Head Start, 
also published, with impacts on children’s social-emotional outcomes ranging from 0.12 to 0.48 
standard deviation. Tools of the Mind has been tested in one randomized control trial, which 
took place in an urban preschool and had an impact on children’s problem behaviors of 0.47 
standard deviation.^'* 

However, because the enhancements had been previously implemented only on a 
smaller scale, little infrastructure existed for supporting larger-scale implementation. Each pro- 
gram had manuals, training materials, trainers, and some form of on-the-ground coaching or 

^^An effect size defines the proportion of a change — specified by change in the standard deviation — that 
is attributable to the intervention. See Box 5.2 in Chapter 5 for more information about effect sizes. See Morris 
et al. (2010); Raver et al. (2008). For the other Incredible Years studies, see Murray, Murr, and Rabiner (2012); 
Reid, Webster- Stratton, and Hammond (2003); Webster- Stratton, Reid, and Hammond (2001); and Webster- 
Stratton, Reid, and Hammond (2004). 

^^There have been three randomized control trials of Preschool PATHS, one of which examined PATHS 
alone (Domitrovich, Cortes, and Greenberg, 2007), a second trial that combined PATHS with a literacy curric- 
ulum (Bierman et al., 2008a; Bierman et al., 2008b), and a third trial of Preschool PATHS plus a professional 
development component (Hamre, Pianta, Mashbum, and Downer, 2012). 

^‘'Bamett et al. (2008). 
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technical assistance, but often there were not enough trainers to start up the program for many 
of the grantees at the same time or the trainers were not easily able to reach grantees in all parts 
of the country. Implementing the enhancements on a large scale to test their effectiveness in a 
manner consistent with the developers’ intent for the appropriate intervention amount and quali- 
ty called for significant supports. To provide in-depth classroom support and technical assis- 
tance, a comprehensive support system, including both training and coaching, as well as tech- 
nical assistance, was needed. In addition, implementation in a research context created further 
challenges that were tied to the Head Start CARES design, which required random assignment 
for different centers within one grantee to different enhancements, a configuration that would 
rarely occur during normal implementation of an enhancement; that is, it is unlikely that differ- 
ent enhancements would be used by the same grantee in a non-study situation. Maintaining fi- 
delity to the design of the enhancements, by making sure that they were implemented fully 
while scaling up, was considered a key goal of the Head Start CARES demonstration, since it 
was cmcial to providing a fair test of the theory of change and the program impacts. 

The effort to maintain fidelity included implementing a professional development model 
with enhancement training by highly skilled trainers who had the proper credentials, as well as a 
standardized coaching component. Both facets of the professional development model were con- 
ceptualized as a way to minimize problematic adaptations and to support the need to bring the 
enhancements to scale quickly. Trainers and coaches played an important role in supporting 
high-quality implementation of the professional development model as well as the teachers’ im- 
plementation of the enhancements in the classroom. A centralized technical assistance team used 
an online management information system to monitor the coaching process and classroom im- 
plementation in real time and to systematically collect implementation data. Technical assistance, 
consisting of data-based quality monitoring and improvement and support of coaches and grant- 
ees, was provided by an MDRC-based technical assistance team working with the enhancement 
developers and trainers. Developers, coaches, trainers, and the technical assistance team worked 
together to integrate the enhancements into the existing Head Start context within each center. 

Given the necessity for such support in scaling up these enhancements, the Head Start 
CARES demonstration falls between an efficacy trial (which, as defined by the Institute of Edu- 
cation Science, is designed to evaluate whether or not a fully developed intervention is effica- 
cious under limited or ideal conditions) and an effectiveness study (which is designed to evalu- 
ate whether a fully developed intervention that has evidence of efficacy is effective when 
implemented under typical conditions through an independent evaluation). The level of im- 
plementation support was less than what might be seen in small, tightly controlled efficacy stud- 


^^See University Council for Educational Administration (n.d.) for the Institute of Education Science’s 
definition of an “efficacy study” versus an “effectiveness study.” 
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ies, but it extended beyond what is typical when preschool centers take on a new initiative.^'’ 
Along with the complexities of the research design, the existing capacity of developers to sup- 
port implementation of the enhancements at the scale required by Head Start CARES was not 
always sufficient. Given the number of trainers available, coaching and technical assistance 
were needed to ensure strong classroom implementation and a fair test of effectiveness. 


Overview of the Report 

The remainder of the report is organized as follows: Chapter 2 describes the enhancements in 
detail and the support system that was put in place to strengthen implementation. Chapter 3 fo- 
cuses on the findings that were observed about the professional development model — namely, 
the dosage (that is, the frequency and duration) and quality of the training and coaching that 
were provided. Chapter 4 discusses classroom implementation of the three enhancements and 
factors contributing to implementation, such as teacher and organizational characteristics. Chap- 
ter 5 addresses whether implementation resulted in changes in teacher practice in participating 
classrooms. Chapter 6 provides an overview of the findings and lessons from this implementa- 
tion study of the Head Start CARES demonstration. 


^^For example, see Odom (2009) and Wise, da Silva, Webster, and Sanson (2005). 
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Chapter 2 

The Head Start CARES Intervention: Enhancements and 
Support for Implementation 


This chapter describes the three social-emotional program enhancements that were tested in the 
Head Start CARES demonstration: The Incredible Years Teacher Training Program, Preschool 
PATHS (Promoting Alternative Thinking Strategies), and Tools of the Mind — Play. Each en- 
hancement has a distinct theory of change for affecting children’s social-emotional competen- 
cies through teacher practices. Each enhancement also has evidence, from small-scale random- 
ized control trials, that it has improved children’s social-emotional outcomes. 

The chapter also provides an overview of the extensive support system for “scale-up” 
— or expansion of the intervention — that was put in place to increase the likelihood of high- 
quality implementation. The supports included a professional development model, comprising 
year-long training and weekly coaching for teachers, as well as ongoing data monitoring and 
technical assistance. The chapter concludes with a presentation of the theory of change that 
guides the implementation research efforts in the Head Start CARES demonstration and serves 
as a road map for the balance of this report. 


Social-Emotional Enhancements Tested in Head Start CARES 

The following sections and Table 2.1 describe each social-emotional enhancement that was 
tested in Head Start CARES and its primary components, as well as vignettes of how key com- 
ponents of the enhancements might be used in the classroom. In addition, information is provid- 
ed on adaptations of those components that were implemented for the Head Start CARES 
demonstration. Some of the adaptations occurred at the outset of the demonstration, before the 
enhancements were implemented with participating centers, while others occurred as the study 
progressed. 

The Incredible Years Teacher Training Program 

The Incredible Years Teacher Training Program is a research-based classroom man- 
agement program that focuses on strengthening teachers’ use of classroom-wide positive man- 
agement and discipline practices, and promoting children’s social and emotional competence in 
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Table 2.1 

Head Start CARES Program Enhancements 


Enhancement 

Description 

Primary Components 

In Practice 

Incredible Years 

Promotes positive 
teacher/child relationships, 
evidence-based classroom 
management and coaching 
strategies, and teacher/ 
parent partnerships. 

Positive attention, coaching 
methods, proactive 
discipline, and other 
behavior supports; 
structured behavior 
management plans; 
relationship building and 
parent involvement. 

Positive attention: “I really 
like the way is sitting 

with her hands in her lap.” 

Preschool PATHS 

Enhances social-emotional 
development through 
explicit lessons and a set of 
generalized teaching 
strategies. 

Daily exchange of 
compliments; weekly 
lessons that promote 
emotional knowledge, 
vocabulary describing 
feelings, self-regulation, 
positive peer relations, and 
problem-solving. 

“How do you think 
felt when gave him a 

hug? That’s right, he felt 
happy! How do you look 
when you feel happy? What 
makes you feel happy?” 

Tools of the Mind 

Focuses on mature make- 
believe play and specific 
learning activities to foster 
cognitive, self-regulation, 
and executive functioning 
skills. 

Make-believe play 
planning, make-believe 
play, and make-believe 
play practice. 

Sample child play plan: “I 
am going to be the bus 
driver.” Teacher supporting 
play: “What route will your 
bus take?” 


the classroom in the context of positive teacher-child relationships. ' Originally designed to ad- 
dress the needs of children with high levels of problem behaviors, The Incredible Years has 


'There have been two randomized trials of Incredible Years Teacher Training alone and two randomized 
trials of Incredible Years along with intensive classroom consultation and stress management training. The 
results across the four studies suggest that the teacher training program is associated with improvements in 
children’s behavior regulation and with improvements in teacher classroom management skills. See Murray, 
Murr, and Rabiner (2012); Reid, Webster-Stratton, and Hammond (2003); Webster-Stratton, Reid, and Ham- 
mond (2001); Webster-Stratton, Reid, and Hammond (2004). CSRP (formerly known as the Chicago School 
Readiness Project) and the Foundations of Learning demonstration both tested Incredible Years Teacher Train- 
ing in conjunction with clinical classroom consultation and stress management training, and both led to im- 
provements in children’s executive functioning (self-regulation abilities that underlie approaches to learning 
through a combination of attention, inhibitory control, and short-term memory skills) and teachers’ classroom 
management. CSRP is not associated with The Chicago School®, which is a trademark of The Chicago School 

(continued) 
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had large impacts on children’s conduct problems (that is, acting-out behavior^) and attention- 
deficit/hyperactivity disorder (ADHD).^ The basic framework for these programs is drawn 
from cognitive social learning theory, which posits that learning occurs within a social con- 
text, at least in part by observing others (such that children learn not only when they are en- 
couraged with positive attention, but also when their peers observe them receiving this atten- 
tion from the teacher). The primary focus is therefore on training teachers how to directly 
promote children ’s adaptive social behavior and how to reduce their problem or inappropri- 
ate behaviors. The approach includes strengthening of positive teacher-student relationships; 
proactive classroom organization (mles and predictable routines); clear and consistent limit- 
setting; social and emotional coaching; praise and incentives to motivate students to learn; 
and proactive discipline and strategies. 

Table 2.2 describes the core components of a Head Start CARES Incredible Years 
classroom. An Incredible Years preschool classroom is characterized by a warm, supportive 
environment where the teachers use classroom management practices, instead of scripted les- 
sons, to promote a positive environment. For example, an Incredible Years classroom has five 
clear mles that children can easily count on their hands; the schedule is posted on the wall; and 
transitions between activities are predictable. The teachers also work to build relationships with 
students by being warm, having fun, and using positive greetings and incentives. When children 
misbehave in the classroom, the teachers ignore them, distract them, and redirect their attention 
to create opportunities to praise positive opposites, and to model to other children how to re- 
spond to negative behavior. Similarly, teachers deal with negative behaviors by illustrating con- 
sequences for those behaviors and having the children take time away (by going into a comer, 
for example) to calm down and practice their breathing. Children with challenging behaviors 
are given behavior plans that involve specific strategies to regulate their behavior and for the 
teachers to guide them in the process. Children are also motivated by incentives such as special 
privileges or rewards for positive behaviors. Teachers support positive behaviors with praise 
and model positive coping strategies for children — such as breathing out or taking time to calm 
down — by enacting them with other adults in the classrooms. Collectively, these changes in 
teachers’ skills in emotional and behavioral management in the classroom are intended to en- 
hance the classroom environment by improving the relationships between teachers and children 
and reducing down time (when no activities are occurring) and transition time (from activity to 
activity), leading to fewer behavioral challenges in children. 


of Professional Psychology. See Morris et al. (2010); Raver et al. (2008). More information about Incredible 
Years can be found at www.incredibleyears.com. See Incredible Years Teacher Training Programs (2012). 
^Acting-out behavior includes aggression, hostility, or opposition. 

^Webster-Stratton, Reid, and Stoolmiller (2008). 
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Table 2.2 

Key Components of Incredible Years in Head Start CARES 

Component 

Description 

Promoting positive relationships 

Relationships are the foundation of Incredible Years, and teachers 
are encouraged to spend time building rapport and trust with each 
child in the class. Children are also encouraged to express their 
emotions with their peers (“feeling talk”). 

Praise and incentives 

When a child demonstrates a positive behavior, teachers use a set 
of strategies, such as rewards and praise, to encourage that 
behavior through positive attention. 

Effective limit setting 

Teachers are trained to ignore, redirect, distract, and use other 
techniques to limit aggression and disruptiveness in the classroom 
and to maximize self-regulation. 

Consequences 

When children do not respond to the strategies above, teachers 
have the option of developing consequences for behavior, which 
can include time out or a loss of privileges. 


SOURCES: Reid, Webster- Stratton, and Hammond (2003); Webster- Stratton, Reid, and Hammond 
(2004). 


Box 2.1 provides a vignette of a teacher using Incredible Years practices in a Head Start 
CARES classroom. A number of key components of Incredible Years are demonstrated in this 
story. First, the teacher uses “persistence” coaching methods to support the child’s willingness to 
wait (waiting behavior); then she uses “proximal praise,” which entails giving attention to posi- 
tive behavior that another child is exhibiting; and then she praises the other child’s willingness 
and ability to share (sharing behavior). Second, she models language to both children to help 
them manage the interaction more effectively. Finally, she follows through with her promise. 

At the start of and during the demonstration, some Head Start CARES grantees ex- 
pressed concerns that some of the Incredible Years practices — for example, the use of food to 
reward positive behavior — were inconsistent with Head Start guidelines about not providing 
food outside of meals. In response to these early concerns, the Incredible Years program made 
slight modifications to the way in which trainers presented these subjects to Cohort Two. (Re- 
call that the demonstration included two cohorts, one in 2009-2010 and one in 2010-2011.) The 
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Box 2.1 

Praise and Incentives in The Incredibie Years 

A teacher sits with two children on the floor. Timmy wants one of the tmcks that the other 
child, Jose, has. Timmy reaches for the truck and the teacher prompts him by saying, 
“Timmy, can you say to Jose, ‘Can 1 have your truck?’” 

Jose responds, “Later.” 

The teacher then says to Timmy, “1 think you can wait 3 minutes.” She shows with her 
three fingers. Then to Jose she says, “In 3 minutes, it will be Timmy’s turn with the truck.” 
Timmy sits on his hands watching and the teacher says, “Wow, Timmy, you are using your 
strong waiting muscles. Would you like to do something else while you are waiting?” 
Timmy looks like he wants to reach for the buck again and the teacher turns to a third child 
and says, “Look at Timmy waiting his turn; he is being such a good friend waiting.” After 3 
minutes have passed, the teacher prompts Jose to give Timmy a turn with the truck, and she 
praises Jose as he does so by saying, “Jose, you are also a good fiiend by sharing your truck 
with Timmy.” She then helps Jose find another toy to play with. 


NOTE: This vignette is based on Incredible Years Teacher Training videos, available at 
www.incredibleyears.com/program/teacher.asp. 


trainer helped teachers understand that the Incredible Years principle of rewarding students in- 
volves finding a fun reward that will motivate the student. The trainer helped teachers share and 
create some possible rewards they could use in their classrooms. Another grantee expressed 
concern about the appropriateness of using “time out” in Head Start. The Incredible Years train- 
er responded by demonstrating how to effectively use time out to help a child calm down and to 
reffame it as “cool-down time” or “time away to think and calm down.” 

Preschool PATHS 

The Preschool PATHS program is a research-based, social-emotional learning en- 
hancement that includes explicit lessons and teaching strategies designed to teach children about 
emotions and responses to peers in social interactions. PATHS is built on research into the im- 

‘^Ibiblished studies of Preschool PATHS include one randomized control trial of PATHS alone, one ran- 
domized control trial of PATHS plus a literacy component called REDI (Research-based, Developmentally 
Informed), and one randomized control trial of PATHS plus a professional development component. The re- 
sults suggest that the program led to improvements in direct assessments and teacher ratings of children’s so- 
cial problem-solving, emotion knowledge, and behavior regulation. See Bierman et al. (2008a); Bierman et al. 

(continued) 
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portance of “emotion knowledge,” self-regulation, and problem-solving skills for children’s 
school readiness and healthy development. The ability to identify and communicate about emo- 
tions (emotion knowledge) helps children accurately interpret the social cues of others, and it 
lays the foundation for effective problem-solving. The primary focus of the PATHS interven- 
tion is to help children develop and internalize social and emotional skills. PATHS helps teach- 
ers create a supportive classroom environment and interact with their students in ways that help 
the children understand and apply the skills that they are taught in the lessons. 

Before the Head Start CARES demonstration began, the Preschool PATHS curriculum 
was adapted from a version that had been developed for elementary school, which targets stu- 
dents in kindergarten through fifth grade. The adaptation process was conducted in part through 
a Head Start University Partnership grant from the Office of Head Start, and the program was 
developed and tested in the Head Start environment. In preschool classrooms, PATHS has re- 
vealed moderate impacts on children’s social interaction.^ 

PATHS lessons are taught to large and small groups, depending on the make-up of the 
class. Lessons cover a range of topics, including complimenting others, understanding feelings, 
stopping and calming down, and problem-solving. Teachers extend and generalize the lessons 
with “extension activities” (such as singing or doing artwork), which provide an opportunity to 
revisit lesson topics and are conducted at least once a week. The teaching strategies that com- 
plement PATHS lessons include emotion coaching, the support of self-regulation, and talking to 
children about social problems, such as what to do if a friend grabs your toy. The teaching strat- 
egies are designed to be used daily to provide children with opportunities to practice skills. 

Table 2.3 summarizes the core components of a PATHS classroom in Head Start 
CARES. A PATHS classroom is characterized by explicit scripted activities and lessons that 
help children recognize and appropriately respond to their emotions. Each day, a PATHS “Kid 
of the Day” is identified; the child is given a special job and receives compliments from the 
teachers and the other children. Classroom mles are posted and proactively reviewed. Teachers 
also display visual reminders of the key curriculum concepts, such as hanging posters around 
the classroom. Teachers who implement PATHS with fidelity to the model use the teaching 
strategies daily to create a positive classroom environment. The teachers describe their positive 
expectations, praise positive models (for instance, when children compliment each other), and 
point out the interpersonal consequences of various kinds of behavior (for example, when 
friends are happy if they share toys). The teachers also learn to anticipate emotionally difficult 


(2008b); Bierman et al. (2010); Bierman et al. (2013); Domitrovich, Cortes, and Greenberg (2007); Domitro- 
vich et al. (2010); and Hamre, Pianta, Mashbum, and Downer (2012). More information about Preschool 
PATHS can be found at www.pathstraining.com. See Preschool PATHS (2012). 

^Domitrovich, Cortes, and Greenberg (2007). 
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Table 2,3 

Key Components of Preschool PATHS in Head Start CARES 


Component 

Description 

Weekly “circle time” lessons Weekly lessons introduce feelings and PATHS strategies to the classroom. 

Puppets of the PATHS characters, stories, and illustrations are used to 
deliver the lessons. 

“Kid of the Day” 

Every day, teachers choose a child at random to be Kid of the Day. The 
class practices giving compliments to this child, which are compiled by 
the teachers and sent home to his or her parents. 

Extensions 

A variety of extension activities are used to further incorporate the 
PATHS lessons into the classroom. These activities can be related to 
music, art, literacy, mathematics, etc. 

The “turtle” technique 

“Doing Turtle” is presented to children as a way to stay in control when 
they are feeling strong emotions. It combines a gesture (crossing the 
arms), controlled breathing, and verbalizing emotions and issues. (“The 
problem is and it makes me feel ”) 

FREE strategies 

Teachers use these strategies to provide children with feedback and 
support to help them reflect on their behavior and self-correct. They are 
used as an alternative to direct commands when children exhibit mild 
forms of disruptive behavior. 


SOURCES: Bierman et al. (2008a); Bierman et al. (2008b); Domitrovich, Cortes, and Greenberg 
(2007). 


situations and prepare children proactively, providing an opportunity for children to self-correct 
when they have inappropriate responses before the teacher exerts more control. Teachers treat 
conflicts as opportunities for learning, actively encouraging the children to be empathetic and 
to try to understand the perspective of someone else, providing feedback, pointing out conse- 
quences, and helping children to problem-solve and enlist the help of their peers. 

The teaching strategies are also designed to help generalize the curriculum concepts. In 
PATHS classrooms, teachers work with children to identify and label children’s emotional ex- 
periences, using a rich emotional vocabulary and modeling or demonstrating self-regulation 
techniques. When children need to calm down, for example, a teacher might prompt a child to 
engage in “turtle,” an emotion regulation technique in which the children cross their arms, con- 
trol their breathing, and articulate their emotions by saying, “The problem is and it makes 

me feel .” The teachers also foster a sense of community by engaging children in conversa- 
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tions, being physically and emotionally available to them, being sensitive to their needs, and 
communicating genuine caring. Finally, teachers encourage children to talk about their emo- 
tions and provide verbal and physical support when children express themselves. 

Box 2.2 provides a vignette of Preschool PATHS in the classroom. As illustrated in the 
vignette, the primary focus of PATHS is children’s understanding of emotions, which is theo- 
rized to be a necessary developmental prerequisite to the internalization of self-regulation, em- 
pathy, and pro-social behavior. Such a developmental approach allows teachers to support chil- 
dren at different stages and in a variety of contexts by “labeling” emotions (that is, giving emo- 
tions a name when observed) and helping children learn appropriate labels and emotional ex- 
pressions, as well as by helping children understand emotions in the context of social interac- 
tions, the management of their own feelings, and problem-solving. 

Preschool PATHS was not substantially altered for the Head Start CARES demonstra- 
tion, in large part because of its genesis within Head Start. After the first cohort of grantees had 
implemented PATHS, however, the PATHS developers more clearly articulated the expectations 
regarding PATHS fidelity — a minimum of one lesson and one extension activity per week. 


Box 2.2 

“Feelings” Lesson in Preschool PATHS 

The lead teacher sits in a circle with the whole class and says, “Let’s play a game and 
look at a drawing of faces. 1 want you to tell me how the people in the picture are feeling. 
We can learn how someone is feeling on the inside by looking for clues. What parts of our 
faces show we are happy?” 

A few children call out “smile” and “mouth.” 

“That’s right,” says the teacher. “Now let’s talk about how mouths look when someone 
feels happy.” The teacher holds up drawings of children and adults and asks the class to 
identify which people are happy and which ones are not. She then asks them to explain 
how they know the people in the drawings are happy. She draws attention to the eyes, 
ears, and noses, and asks the class to tell her if there is a difference between happy and sad 
eyes, ears, and noses. The teacher then hands out blank face templates and pictures of 
mouths, eyes, ears, and noses. She asks the children to make a happy face using the pic- 
tures and then asks them to make a sad face. 

Later in the day the children are playing in play areas. The teacher walks around, calling 
attention to children who are happy, noting the facial cues that show how they feel. She 
asks one or two children to describe how other children feel by asking them to explain 
how they know what the child feels. 
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Tools of the Mind 

Tools of the Mind is a curriculum based on the theories of Russian psychologist Lev 
Vygotskian,*’ which was developed for preschool and kindergarten classrooms/ This program is 
based on the belief that, particularly in early childhood settings, a focus on interpersonal (peer- 
to-peer) interactions is the most beneficial approach for introducing young children to their 
“tools of the mind.” 

This program targets cooperative play skills, self-regulation, and social problem- 
solving. Additionally, there is an extensive emphasis on fostering cognitive self-regulation (ex- 
ecutive functioning) skills and related approaches to learning.* Executive functioning, a compo- 
nent of self-regulation, in early childhood includes working memory (or the ability to keep a 
number of pieces of information in the mind at once), set-shifting (or the ability to flexibly shift 
between pieces of information), and inhibition (or the ability to stop or repress an immediate 
response). Rather than providing explicit lessons on these skills, however. Tools of the Mind 
changes the way that make-believe (pretend) play and other learning experiences are stmctured 
and supported in the classroom. Teachers focus the classroom on a specific theme for a few 
weeks, incorporating it into all activities, and all the Head Start play areas use that theme — 
such as jobs in the neighborhood or people in the family — to make children’s play richer and 
more interactive. A central component of Tools of the Mind in preschool is a daily 50-minute 
block of time that is devoted to make-believe play, which teachers organize in very specific 
ways to enhance children’s planning skills, understanding of social roles, memory and focused 
attention, and social-emotional understanding. In addition, the program includes self-regulation 
games and restmctures “circle time” (a large-group activity period) to have less waiting time 
and fewer activities, and uses literacy, math, and science learning activities, to encourage chil- 
dren to take control of their own learning and support each other’s learning.'® 

A previous experimental study that evaluated Tools of the Mind found moderate 
treatment effects for behavioral problems as reported by teachers." An additional study 


®Vygotsky believed that children use interpersonal communication to learn and internalize tools that lead 
to higher mental functions and allow them to take charge of their own learning. 

^More information about Tools of the Mind can be found at www.toolsofthemind.org. See Tools of the 
Mind (2013). 

^Previous studies, such as the Foundations of Learning demonstration, have found that social-emotional 
interventions can improve children’s executive functioning and other approaches to learning. See Morris et 
al. (2010). 

®In make-believe, or pretend, play, children use their imaginations to role-play, pretend they are different 
characters, play out stories, and enact scenarios that rely on and encourage creativity. 

'"Teachers were also expected to change themes numerous times over the course of the year, which re- 
quired creating multiple sets of props with which the children could play. 

"The findings of one randomized control trial of Tools of the Mind have been published. The results sug- 
gest that the program led to improvements in teacher ratings of children’s behavior at school. See Barnett et al. 

(continued) 
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found that Tools of the Mind had a positive effect on children’s executive functioning, spe- 
cifically inhibition. 

Table 2.4 describes the core components of a Tools of the Mind classroom in Head 
Start CARES. A Tools classroom focuses on mature make-believe play as a way to foster self- 
regulation, as well as cognitive and executive functioning skills. In a Tools classroom, the 
teachers move with ease from student to student during play planning, “scaffolding” the activi- 
ty, with students working at all levels. Scaffolding is a practice that teachers use to support a 
child’s activity or response at his or her current level of understanding while extending the ac- 
tivity or response in order to push the child to the next level of ability. For example, a child may 
not yet know how to spell but may know the alphabet. A teacher would “scaffold” this child by 
telling her the letters in a word and letting her write them down until the child can write simple 
words by herself Some children may be focused on “making a plan” by drawing on their own, 
while others might be writing in the parts of words that they are able to sound out. The teachers 
encourage children to talk and work together, and potential conflicts are worked out at the table 
before children leave to play. 

A particular play theme is selected and represented in multiple play areas that have an 
abundance of child-made props, such as chef hats, “food” made from Play-Doh®, and hand- 
written menus. Teachers circulate throughout the room, scaffolding children during the daily 
50-minute play block, and as many as half of the children are deeply engaged and stay in a role 
or scenario with peers for an extended period of time. The teachers participate in make-believe 
play practice and create inventive and engaging scenarios; for example, one teacher might pre- 
tend to be a customer in a restaurant, while the other teacher takes the role of a waitress who 
spills the drinks. “Fingerplays” or “pretend transitions” (such as “This Fittle Piggy” or pretend- 
ing to be “quiet as a mouse”) are used during all or most transitions between activities 
throughout the day, and all or most students are able to focus their attention quickly and remain 
regulated during transitions. The teachers use different pretend transitions throughout the day, 
and there are no extended large-group activities or wait times between activities without active 
engagement. 


(2008). Since then, the findings (unpublished) from two randomized control trials of Tools and one randomized 
control trial of Tools in conjunction with a math curriculum have failed to show statistically significant differ- 
ences (that is, differences that have a high probability of being attributable to the program) in literacy, lan- 
guage, and math skill acquisition or in executive functioning for children in Tools (or Tools plus math) class- 
rooms compared with standard preschool programs. See Clements, Sarama, Unlu, and Layzer (2012); Farran, 
Lipsey, and Wilson (2012); and Lonigan and Phillips (2012). 

'^Diamond, Barnett, Thomas, and Monro (2007). 


22 



Head Start CARES Demonstration 


Table 2,4 

Key Components of Tools of the Mind in Head Start CARES 


Component 

Description 

Play planning and scaffolded 
writing 

Teachers support all students while developing their play plans. A play 
plan might include the child’s name, an illustration of the plan showing 
what the child will do during play time, lines representing the words of 
the play plan, and starting/beginning sounds to each word. 

Make-believe play 

Teachers set aside a required block of time for play with a clear theme. 
Teachers circulate the room scaffolding play and encouraging deep 
child engagement in play. 

Make-believe play practice 

Teachers create inventive and engaging scenarios that are meant to be 
carried over by the children into their own play. This activity is meant 
to have a playful feel with high levels of student participation. 

Attention-gathering activities 

Fingerplays or pretend transitions are used as attention-gathering 
activities during transition times. Fingerplays are games or songs for 
which the fingers are used, like “This Tittle Piggy.’’ Pretend transitions 
are transitions in which a “pretend” element (such as pretending to be 
“quiet as a mouse”) is used. 


SOURCE: Barnett, Yarosz, Thomas, and Hombeck (2006). 


Box 2.3 provides a vignette of Tools of the Mind in the classroom. As described, chil- 
dren are asked to plan their play before they engage in it, and teachers are asked to scaffold that 
play and learning for children. Moreover, much of the learning takes place in extended pretend- 
play sequences, in which children regulate their own behavior as well as that of their friends to 
fit within the planned play sequence. In some ways, of the three enhancements. Tools of the 
Mind required the most from teachers — both in terms of their ability to match their response to 
children’s skill levels and to effectively support children’s play planning and pretend play. In 
addition, given the number of activities and the way in which the day is stmctured. Tools of the 
Mind required a high level of coordination between the lead and assistant teachers. As such, it 
was considered the most complex of the three enhancements. 
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Box 2.3 

Make-Believe Play Planning in Tools of the Mind 

The lead teacher is working with students who are seated at one table, and the assistant teach- 
er is working with students at another table. Each child has a half-sheet of paper, with pictures 
or lines on some of their papers and lines with letters on them on others. The lead teacher 
works with one student, Ashley, to help her plan and develop her play scenario before she be- 
gins. 

Pointing beneath each line, Ashley says, “1 am going to train.” 

“Wonderful — 1 can see that in your picture!” says the teacher as she points to Ashley’s 
drawing of herself sitting on a train in her play plan. “Now, all you need to do before going to 
the train station is to add in some more letters.” Because it is half-way through the year and 
they have been practicing all year, Ashley is able to write “1 am going to” on her own. The 
teacher and child work together to spell out train, with the teacher scaffolding Ashley’s letter 
knowledge as necessary. 

The teacher then asks, “What route will the train take?” 

“1 want to take the other kids to school,” the child says. 

The teacher helps Ashley extend the play scenario by responding, “Wow! You are a great 
train conductor. You know exactly how to get the passengers to school. Yesterday, we 
learned how to get all your passengers on the train and go ‘All aboard! ’ Let’s practice!” 

The child goes to the train station center, makes the “All aboard!” gesture, and begins to play 
the “train” game with her peers, each taking on a different role in the activity. 

In taking on pretend roles, Ashley and her friends are learning how to take on other perspec- 
tives, which is critical to social interaction, as well as strengthening their cognitive flexibility 
and inhibition skills by switching between “themselves” and their “pretend role” without 
switching back out of the pretend role or moving on to play something else. 


In the typical implementation of Tools of the Mind, teachers conduct the comprehen- 
sive, full-day Tools model over a two-year period. Because this curriculum extends across mul- 
tiple content areas, components are phased in over time to allow teachers to gradually imple- 
ment the full curriculum. To accommodate the structure and time frame of the Head Start 
CARES demonstration, Tools developers adapted the curriculum to allow for implementation in 
one year, meaning that Tools was condensed into a one-year enhancement rather than a two- 
year curriculum designed to structure all components of the school day. The developers, how- 
ever, maintained the traditional Tools emphasis on Vygotskian concepts and focused on make- 
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believe play planning (done by the children), make-believe play, and the teacher’s make-believe 
play practice for the whole classroom. 


Implementation Support System 

A key challenge of many large-scale trials is maintaining fidelity to the intervention model as 
designed. In previous efficacy trials of early childhood interventions involving a relatively small 
number of classrooms, the developers of the intervention being tested were primarily responsi- 
ble for monitoring and responding to implementation issues, and were able to respond in a high- 
ly individualized manner. While this arrangement provided support for each unique implemen- 
tation challenge that arose, it was not useful for identifying a more systematic approach to as- 
sessing, categorizing, and addressing implementation challenges at scale when implementation 
was more removed from the developers. 

Therefore, considerable attention was paid to ensuring that the three evidence-based en- 
hancements in Head Start CARES were supported by a strong and systematic framework, de- 
picted in Figure 2.1, when scaling up for the demonstration. As explained in Chapter 1, Head 
Start CARES involved 17 grantees spread out across 10 states around the country. As part of the 
research study, each grantee was required to implement three separate enhancements at the 
same time — an approach that is likely to be more burdensome than introducing only one new 
intervention per grantee, which is a more typical scenario. Initiation and implementation of 
three enhancements at the same time created an extra hurdle to overcome, potentially leading to 
less support and buy-in from grantee administrators. 

As discussed below, given the need to scale up rapidly, a more stmctured and intensive 
support system for achieving fidelity was needed. This included (1) prepared enhancement ma- 
terials, including manuals and various teaching props; (2) professional development, including 
training by certified trainers and classroom-based coaching; and (3) monitoring and technical 
assistance. 

Prepared Materials for Trainers, Coaches, and Teachers 

Trainers, coaches, and teachers used enhancement-specific materials that developers al- 
ready had written and put in place. These previously developed manuals made up the founda- 
tion for supporting teachers while they implemented the enhancements in the classroom, provid- 
ing lists of needed materials, scripts, and explanations for activities. 
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Head Start CARES Demonstration 
Figure 2.1 

Program Model Components 



Lead and Assistant 
Teachers 



Professional Development 

Training 

The Head Start CARES training component included well-developed pre-existing cur- 
riculum manuals and training materials, provision of the appropriate training sessions to lead 
and assistant teachers, and the trainers’ support of classrooms through classroom visits. Training 
took place both on- and off-site, and some sessions took place during “professional develop- 
ment days” that were set aside for that purpose, while others occurred during the week or on 
Saturdays. 

Although the foci of the teacher training varied by enhancement, the structure for the 
delivery of content was similar across all three enhancements. Both lead and assistant teachers 
participated in a series of training sessions over the course of the academic year, with the train- 
ing providing pedagogical and theoretical background about how to understand and implement 
the enhancements’ practices and activities. The Incredible Years offered six days of training; 
Tools of the Mind offered five days; and Preschool PATHS offered four days. Preschool 
PATHS and Tools of the Mind provided new content in all the training sessions except for the 
final one, which was used as an opportunity to review and reinforce content from the previous 
training sessions. The Incredible Years, however, presented new content in all the training ses- 
sions, including the final one. 

The trainers for Head Start CARES were certified or eligible for certification to deliver 
teacher training by the respective developers. The scaling up required that a large number of 
new trainers be certified to lead coaches and teachers in training. Due to the need to scale up 
rapidly, a number of trainers completed the certification processes during the implementation 
years. To become certified. Incredible Years trainers needed to be accredited or educated as a 
teacher, psychologist, or school counselor and attend a workshop for trainers, after which they 
needed to conduct Incredible Years teacher training and receive positive teacher evaluations and 
satisfactory feedback from a certified trainer. PATHS trainers were required to have at least two 
years of experience with the enhancement, attend a trainer’s workshop, and receive ongoing 
training in PATHS delivery to be certified. The Tools certification process required participants 
to be certified as eligible by developers. 

Coaching 

Despite the existing pool of trainers, enhancement developers did not have the number 
of certified trainers needed to serve in a supportive role for teachers on a weekly basis and 
across multiple Head Start centers. In-classroom coaching was therefore viewed as an essential 
component of the professional development model for all three enhancements and was concep- 
tualized as a way to augment and reinforce the training content as well as to standardize the 
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support provided to teachers for implementation across the enhancements. Although all the en- 
hancements had some level of in-classroom consultation embedded in them before Head Start 
CARES began, the research team collaborated with developers to create one standard coach- 
ing component that was used across all enhancements for replication and scale-up purposes. 
The coaching component in Head Start CARES was conceptualized as teacher-focused, collab- 
orative, instmctional, evaluative, and nonsupervisory, and was designed to help teachers use the 
enhancements with fidelity in their classrooms. 

In general, in-classroom coaching complemented the training by providing a fomm for 
teachers to experience the blending of the enhancement theory, content, and practice, and to 
receive ongoing feedback in order to become competent in the enhancement and sustain chang- 
es in their instmctional practices. Coaching lead and assistant teachers was viewed as essential 
for strong classroom implementation and had the added benefit of ensuring that a teacher who 
was competent in implementing the enhancement would always be available in the case of 
teacher turnover or long-term absences. 

Grantees hired coaches with funds provided by the study. Coaches were responsible for 
an average of 4, and as many as 13, classrooms each. The coaching component of the profes- 
sional development model required coaches to spend a total of 90 minutes in the classroom each 
week, including a 30-minute meeting with teachers and a 60-minute observation period. The 
coaches also participated in regular supervisory phone calls with their trainers to discuss imple- 
mentation progress and challenges. In addition to their coaching responsibilities, coaches pro- 
vided logistical and implementation support, such as finding materials or helping to facilitate 
discussions, for the delivery of the enhancement-specific training sessions in coordination with 
the trainers and grantee liaisons (a staff person designated by the grantee, who sometimes su- 
pervised coaches and teachers). 

The expected weekly commitment for coaches was an average of three hours per class- 
room, as coaches were also responsible for completing online logs that were used for monitor- 
ing implementation and collecting data, as explained in the next section. 


'^Previous trials of the three enhancements varied in the target (or focus), frequency, and modes of consul- 
tation. In The Incredible Years, the trainer was available to answer questions from the teachers, and coaching 
was also provided for implementation of a separate child component, called Dinosaur School. In Preschool 
PATHS, trainers supervised implementation with phone calls and periodic face-to-face meetings. In Tools of 
the Mind, coaches provided weekly in-classroom meetings. 
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Monitoring and Technicai Assistance 


Monitoring 

An important tool for monitoring implementation was the Head Start CARES manage- 
ment information system (MIS), which served as a repository for coaches and trainers to submit 
information about how teachers were implementing the enhancements and how often coaches, 
teachers, and trainers interacted. The MIS allowed for “real-time” monitoring (as opposed to 
intermittent or retrospective monitoring) of both the coaching process and progress on class- 
room implementation. 

The system was designed to support technical assistance, management, and monitoring 
of fidelity to the enhancement design as the enhancement was delivered in the classroom. It in- 
cluded user-friendly, online “coach weekly logs” and “coach monthly fidelity logs,” as well as 
“trainer fidelity logs.” Coach weekly logs captured and documented weekly coaching sessions; 
coach monthly fidelity logs documented fidelity to the enhancement design. The coach weekly 
logs and the coach monthly fidelity logs enabled the technical assistance team to monitor, react 
to, and track the progress of implementation. Coaches completed separate coach weekly logs 
and coach monthly fidelity logs for each classroom with which they worked. In addition to their 
weekly reports about coaching for each classroom and their monthly reports about fidelity to the 
enhancements, the coaches reported on such topics as the teacher-coach relationship lead teach- 
er-assistant teacher relationship. Appendix B presents the questions from the coach weekly log 
and the coach monthly fidelity log, for which coaches provided answers in the MIS. Trainers 
similarly used logs to report on fidelity each time they visited a classroom, to rate coach quality, 
and to report on their contact with coaches each time they spoke. 

As part of the ongoing monitoring of implementation, a technical assistance threshold 
was created on the coach monthly fidelity log for satisfactory implementation. On a scale of 1 
(low) to 5 (high), classrooms with a monthly coach rating at or above a 3, meaning that teachers 
were implementing the enhancement occasionally but not consistently, were considered to have 
reached the threshold for satisfactory implementation. When classrooms were not able to reach 
or dropped below the threshold, technical assistance was provided. The technical assistance, 
described below, included additional support to the coach on classroom implementation chal- 
lenges, outreach to the developers and trainers, or discussions with center directors or grantee 
liaisons if necessary. Coach weekly logs also included open-ended comment boxes in which 
coaches could leave remarks for the technical assistance team about implementation issues as 
they arose. Issues ranged from concerns about resistant teachers to observations about the need 
for supplies or problems with paychecks. This was supplemented by research team and trainer 
visits to Head Start CARES classrooms. Table 2.5 provides an overview of all the data sources 
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Table 2.5 

Head Start CARES Implementation Research Data Sources 


Quantitative 



Qualitative 

Attendance Feedback Coach Trainer 

Teacher 

Classroom 

Research Team 


Domain 

Forms 

Forms 

Logs" 

Logs’’ 

Self-Survey 

Observations" 

Interactions'’ 

Interviews 

Trainine® 









Dosage 

X 



X 




X 

Quality 


X 


X 



X 

X 

Content 


X 





X 

X 

Coachine 









Dosage 



X 




X 

X 

Quality 



X 

X 

X 


X 

X 

Content 



X 


X 



X 

Coach support/supervision 



X 

X 




X 

Coach characteristics 



X 





X 

Classroom imolementation 









Fidelity 



X 

X 



X 

X 

Teacher practices 






X 



Contextual factors 









Child characteristics 








X 

Teacher characteristics 



X 


X 



X 

Classroom characteristics 






X 

X 

X 

Organization characteristics 



X 


X 

X 

X 

X 


NOTES; For a full explanation of all Flead Start CARES data sources, see Appendix C. 

^Coach logs include coach weekly logs, coach monthly fidelity logs, coach end-of-year reflections, and data from the coach demographics 
survey. 

'’Trainer logs include supervision logs and fidelity reports. 

‘’Classroom observations include the Classroom Assessment Scoring System (CLASS) and the Adapted Teaching Style Rating Scale 
(Adapted TSRS). 

‘'Research team interactions include phone calls, e-mail exchanges, and in-person meetings with key players. 

‘’Training includes training sessions and trainer classroom visits. 



that were used for the evaluation. For a full explanation of Head Start CARES implementation 
data sources, see Appendix C. 

Ongoing Technical Assistance 

Finally, MDRC provided technical assistance to all members of the implementation 
support network (which included enhancement developers, trainers, coaches, teachers, and 
grantee liaisons), supported the implementation of the enhancements, and facilitated communi- 
cation. For grantees, the technical assistance team provided a grantee liaison tool kit that de- 
scribed key roles and responsibilities, outlined the implementation schedule, and contained cop- 
ies of coach weekly logs and coach monthly fidelity logs, as well as materials such as draft job 
descriptions and interview guides for the grantees to use when they hired coaches. The technical 
assistance team also helped grantees and coaches plan the logistics of training sessions when 
necessary. Coaches were provided with a tool kit that articulated the coaching plan and related 
activities. For Cohort Two, coaches also participated in cross-grantee coach calls with the tech- 
nical assistance team, which provided an opportunity for coaches who were working with the 
same enhancement to share successes as well as challenges that arose during implementation. 

In order to facilitate communication among the many participants in the study, the tech- 
nical assistance team conducted a three-day kick-off meeting and training workshop on Head 
Start CARES that introduced all key stakeholders in Cohort Two to the demonstration. The 
training workshop was conducted during the summer of 2010, before the enhancements were 
implemented in Cohort Two. Grantee liaisons and coaches from the Cohort Two grantees, de- 
velopers, trainers, and the research team participated in the training. Grantee liaisons attended 
brief orientation sessions on each enhancement. Coaches received training in the coaching com- 
ponent, their specific enhancement, and completion of the coach weekly logs and coach month- 
ly fidelity logs. The technical assistance team also monitored the incoming coach weekly logs 
and coach monthly fidelity logs throughout implementation, using the information to feed back 
to developers and grantees as necessary. This was accomplished through monthly site team dis- 
cussions with grantees and developers. 


Processes for Supporting Implementation: Supervision and 
Communication 

As described above, the demonstration required the support of many people playing comple- 
mentary, and sometimes overlapping, roles. Their key roles and responsibilities are summarized 


'"'site teams, which comprised operations staff from MDRC and MEF Associates, met monthly with a 
designated member of the grantee to check in about implementation. 
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in Table 2.6. Enhancement developers, who had designed each of the enhancements, supported 
and supervised their designated enhancement experts, the trainers. Trainers trained and super- 
vised the coaches, each of whom supported implementation for a specific grantee and was con- 
sidered an expert in how to implement the enhancement in that center.'^ Coaches received en- 
hancement-related supervision from the trainers and developers, and received administrative 
supervision from the grantee that hired them. Each grantee designated a staff person, the grantee 
liaison, who supervised coaches and teachers in various ways, either directly or as part of their 
Head Start CARES responsibilities. 

Similarly, a complex network was needed to facilitate communication across the nu- 
merous key players. The grantee liaisons, who were responsible for sending clear messages to 
all of the staff about implementation expectations, recruiting and hiring of coaches, providing 
administrative oversight for the coaches, and ensuring that teaching staff received support for 
implementation of the enhancements in their classrooms, also directly communicated with 
MDRC. Site teams, made up of MDRC operations staff who were responsible for supporting 
grantees, each focused on a different grantee, holding monthly discussions with the grantee 
and at least one site visit to a grantee to discuss implementation progress and challenges. For 
each enhancement, the technical assistance team assigned a lead contact person who gathered 
information from developers, trainers, coaches, and grantees to address implementation chal- 
lenges. The technical assistance team shared themes from developers, trainers, coaches, and 
grantees to identify issues that cut across grantees and to work with developers and trainers to 
address those issues. 

In summary, implementation and scale-up of training, coaching, and the social- 
emotional enhancements required substantial effort among all the stakeholders who were in- 
volved in the demonstration. Strong implementation of the Head Start CARES enhancements 
required a well-articulated framework for scaling up professional development and support. As 
noted above, all three enhancements had prepared materials for teachers to use in the class- 
room, and classroom implementation was supported by focused training and coaching, as well 
as by continuous data monitoring and technical assistance. A Web-based MIS provided ongo- 
ing information to developers, trainers, and the technical assistance team on the coaching pro- 
cess and classroom implementation. Ongoing monitoring and real-time information from the 
MIS, together with consistent communication among all parties, enabled the Head Start 
CARES team to make adjustments along the way that supported and strengthened implementa- 
tion of the enhancements. 


'^Coaches were hired by one grantee each and assigned to one enhancement each, which in many cases 
meant that they were responsible for only one center but in some cases meant that they served two or three 
centers. 
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Head Start CARES Demonstration 


Table 2.6 


Title, Responsibilities, Employment, and Supervision of Key Players in 

Head Start CARES 


Title 

Responsibilities in Head Start CARES 

Employment and Supervision 

Enhancement developer 

Designed the enhancement, provided 
content-related support to coaches 
and teachers, and supervised trainers. 

Various employment and 
supervision structures (e.g., 
universities, research centers). 

Trainer 

Delivered training sessions to coaches 
and teachers on enhancement content. 

Employed and supervised by the 
enhancement developer. 


Visited classrooms to support coaches 
and teachers with enhancement 
implementation. 



Provided supervision and regular, 
feedback on coaches’ performance 


Coach 

Attended training sessions with 
teachers. 

Received ongoing content-related 
support from trainer and enhancement 
developer. 

Observed and met with teachers 
weekly to discuss enhancement 
implementation. 

Employed by grantee-level 
administrators and center-level 
administrators and directors. 

Supervised by the enhancement 
developer/trainer, grantee-level 
administrators, and center-level 
administrators and directors. 

Lead teachers and 
assistant teachers 

Attended training sessions alongside 
the coach. 

Received ongoing support from 
coaches and trainers throughout the 
year. 

Employed and supervised by 
grantee-level administrators and 
center-level administrators and 
directors. 


Implemented assigned enhancement 
in the classroom. 


Grantee liaison/grantee 
administrators 

Recruited, hired, and supervised 
coaches. 

Employed and supervised by 
other senior administrative staff 


Monitored implementation 
throughout the year. 


MDRC technical 
assistance team 

Provided ongoing technical assistance 
to grantee-level administrators, center- 
level administrators and directors, 
enhancement developers, trainers, 
and coaches throughout the year (e.g., 
monitored log completion, facilitated 
communication between trainer and 
grantee-level administrators). 

Awarded a contract by the 
Administration for Children and 
Families to implement and 
evaluate Head Start CARES. 
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Theory of Change Guiding the impiementation Study 

As shown in Figure 2.2, the Head Start CARES implementation study was guided by a logic 
model framework. As mentioned previously, fidelity to the enhancements as they were intended 
to be used was considered critical to ensuring a fair test. As such, the theory of change hypothe- 
sized that: 

1. Implementation of the enhancements with fidelity to the model would 
strengthen existing practices and/or lead to changes in teacher practice. 

2. Changes in teacher practice would lead to improved classroom interactions. 

3. Improved classroom interactions would lead to improved child outcomes. 

The implementation study focuses on the first of these three steps — implementing the 
enhancements with fidelity and changing teacher practice. Chapters 3 through 5 assess whether 
the enhancements were implemented with fidelity and preview preliminary findings about 
changes in teacher practice. The last two steps of the theory of change are examined in a sepa- 
rate report on the impacts of Head Start CARES.'*’ 

In this theory of change, three facets were deemed central to the process of implementa- 
tion: the professional development model that was created to support implementation on an 

ongoing basis (that is, the training, in-classroom coaching, and technical assistance discussed 
above), teachers’ actual implementation of the enhancements in the classroom, and environ- 
mental or contextual factors that could support or hinder fidelity.'* The professional develop- 
ment model is described in detail above. With regard to classroom implementation, it was ex- 
pected that given a sufficient dosage and quality of training and coaching, program teachers 
would build the necessary enhancement-specific knowledge and skills, view the program as 
acceptable, and be confident and motivated in implementing the enhancements. As shown in 
Figure 2.2, it was expected that the same model might be implemented differently across the 
diversity of contexts in which Head Start CARES operated. That is, environmental and contex- 
tual factors such as the characteristics of the people and organizations involved may have 


'^Morris, Mattera, Castells, and Bangser (forthcoming). 

'Wll three facets focus on lead teachers (instead of both the lead and assistant teachers) and their imple- 
mentation of the Head Start CARES enhancements. This decision was based on the level of resources available 
and is consistent with the companion impact report (Morris, Mattera, Castells, and Bangser, forthcoming). 

'*Head Start traditionally emphasizes parent involvement, and the enhancements did include varied de- 
grees of parental outreach, though fiscal constraints did not allow for allocating resources toward exploring the 
role of parents in implementation. 
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shaped how the enhancements were implemented in the classroom. In the Head Start CARES 
demonstration, these factors included characteristics of teachers, classrooms, and centers, as 
well as characteristics of the broader Head Start grantees. The next chapters address all three 
constmcts — the professional development model, teachers’ implementation of the enhance- 
ments, and contextual factors — and their delivery in the Head Start CARES study. 
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Chapter 3 

Fidelity to the Professional Development Model 


This chapter describes how closely the professional development model, which includes train- 
ing and in-classroom coaching, was followed in the Head Start CARES demonstration. To as- 
sess the fidelity to the professional development model that was achieved during implementa- 
tion, the chapter presents information on the dosage (frequency and duration) and quality of the 
training and coaching. Costs associated with training and coaching for each of the three en- 
hancements are presented in Appendix D. 

On the whole, teacher training and coaching were implemented as intended. Certified 
trainers provided engaging learning materials to teachers throughout the school year, with 
teachers and coaches participating in the training sessions at high rates. Coaches were able to 
meet with teachers often, facilitating the implementation of the enhancements in the classroom 
by developing relationships with the teachers and conducting weekly coaching sessions. Train- 
ers who supervised and supported the coaches indicated that coaches were of moderately high 
quality and helped strengthen classroom implementation. 


Understanding Fidelity to the Head Start CARES Demonstration 
Professional Development Model 

Fidelity is a constmct that assesses the gap between an intended intervention or service and how 
it was actually provided. Fidelity includes the measurement of adherence, dosage, and quality of 
an evidence-based practice or service. ' Fidelity in Head Start CARES focused on two compo- 
nents: implementation fidelity, or fidelity to the professional development model; and interven- 
tion fidelity, or fidelity to the enhancement model, or design, as the intervention was delivered 
in the classroom. This chapter presents findings about the quality and dosage with which the 
professional development model was implemented, as well as the support that trainers gave 
through phone calls and visits to classrooms and coaches. Fidelity to the enhancement design as 
it was delivered in the classroom is examined in Chapter 4. 

Training and coaching were assessed by examining (1) the extent to which they adhered 
to the professional development model as designed; (2) the frequency (number) and duration 
(length) of completed coaching sessions; (3) the quality of coaching and training; (4) participant 
responsiveness to coaching and training; and (5) the amount of support that trainers provided to 
classrooms and coaches. Drawing from the theory of change that is presented at the end of 

'Dusenbiiry, Brannigan, Falco, and Hansen (2003); Lloyd, Supplee, and Mattera (2013). 


37 



Chapter 2, it was hypothesized that if the training and coaching adhered fully to the professional 
development model and were implemented at a high dosage and quality, the likelihood that the 
enhancements would be implemented, and implemented well, in the classroom would increase. 


Training in the Head Start CARES Demonstration 

Fidelity to the Head Start CARES training component was determined by the extent to which 
grantees offered all of the planned training sessions, trainers delivered the full content of the 
training, and lead and assistant teachers attended enhancement trainings together. As such, fidel- 
ity was assessed by examining (1) the dosage of training received; (2) the quality of training 
received; and (3) the support that trainers provided for classrooms. 

Dosage of Training 

A first step in assessing fidelity to the professional development model was understand- 
ing whether training was delivered as intended. As shown in Table 3.1, the planned number of 
training sessions differed across enhancements, with six days of training for The Incredible 
Years, four days of training for Preschool PATHS, and five days of training for Tools of the 
Mind — Play. 


Head Start CARES Demonstration 
Table 3.1 

Head Start CARES Training Sessions Throughout the School Year, 

by Enhancement 


Enhancement 

Aug./Sept. 

Oct. 

Nov. 

Jan. 

Feb. 

Mar. 

Incredible Years 

1 day 

1 day 

1 day 

1 day 

1 day 

1 day 

Preschool PATHS 

2 days 

1 day 


1 day 



Tools of the Mind 

2 days 

1 day 


1 day 


1 day 


NOTES: The table shows an estimated schedule of training sessions for each enhancement. Some 
grantees held training earlier or later than the estimated months listed for each training. 

A day of training lasted between 6 and 8 hours, with time for breaks and lunch. 

Incredible Years presented new content in each training session. Preschool PATHS and Tools of the 
Mind provided new content in the first three and four training sessions, respectively, and reinforced the 
content during the last training session. 
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• Teacher attendance at the training sessions was generally high across all 
grantees and all enhancements. Lead and assistant teachers generally at- 
tended the training sessions together. 

As shown in Table 3.2, training was generally well attended by both cohorts, with 82.4 
percent of all sessions attended by lead and assistant teachers together, 93.5 percent attended by 
lead teachers, and 87.1 percent attended by assistant teachers. Moreover, lead and assistant 
teachers often attended the training sessions together, although that was not achieved uniformly 
across grantees. (That is, attendance by lead teachers was higher than attendance by assistant 
teachers.) Lead teachers were somewhat more likely to attend training than assistant teachers, 
possibly because it was difficult for both teachers to be absent from the classroom at the same 
time, and lead teachers were often sent to represent the classroom. 


Head Start CARES Demonstration 
Table 3.2 

Attendance of Lead and Assistant Teachers at Training Sessions, 

by Enhancement 


Percentage of Sessions Attended 
All Incredible Preschool Tools of 


Attendee 

Enhancements 

Years 

PATHS 

the Mind 

Lead and assistant teachers together 

82.4 

81.8 

83.4 

81.8 

Lead teacher 

93.5 

94.4 

94.2 

91.8 

Assistant teacher 

87.1 

86.6 

88.3 

86.3 

Sample size“ 

230 

77 

77 

76 


SOURCE: MDRC calculations based on the training attendance forms. 

NOTES: Training spanned a full school year, from summer through the following spring. 
“For all variables in the table, data are available for 100 percent of the sample. 


Training of lead and assistant teachers together was conceptualized as a way to support 
implementation of the enhancements. For example, if both teachers were trained, classroom 
dosage would not necessarily be reduced if one teacher was absent or left the classroom, be- 
cause children would still be exposed to the enhancement content by the co-teacher. The re- 
search team also believed that if teachers attended training sessions together, they would have a 
common framework and language to use when discussing classroom issues related to imple- 
mentation of the enhancements. One assistant teacher in Tools of the Mind explained, “[Train- 
ing sessions were] very supportive of what [the lead teacher] told me and then they would elab- 
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orate on it even more.” Teachers indicated that attending training sessions together was helpful 
because it supported their efforts to work together and share the tasks related to implementing 
the enhancements. 

Coaches attended training as well, providing additional support to teachers at the train- 
ing sessions. Aside from an introductory training workshop that was conducted for Cohort Two, 
coaches were trained in the enhancement content alongside teachers. Coaches’ attendance at 
training was generally high as well, with coaches attending, on average, 86 percent of the 
scheduled teacher training sessions. Teachers reported that the coaches’ attendance at the train- 
ing sessions with them was very valuable as a support; however, teachers also wanted coaches 
to be more knowledgeable about the content of the enhancements before they began their actual 
work in the classroom (rather than just learning the content simultaneously with the teachers). In 
other words, teachers wanted coaches to have expertise in the enhancement content ahead of 
time, but they also appreciated having coaches join the training as an added support. 

Training Quaiity 

Fidelity to the professional development model was also assessed by observing how 
well the training was received by participants and the quality of the training sessions. Three sets 
of perspectives on the quality of the training were collected, from the teachers, the coaches, and 
the research team. These multiple perspectives provided a more complete picture of how the 
training was implemented, given that different participants sometimes had differing opinions 
about implementation.^ For instance, teachers and coaches were the primary participants in 
training, with teachers experiencing the training as it was actually delivered and coaches playing 
a logistical and supportive role in training as well as attending the training. In comparison, the 
research team was not the target of training and thus had a more removed viewpoint. Taken to- 
gether, these perspectives provide insight about the quality of the training for teachers and 
coaches. 

• Training quality was generally strong across all grantees and all en- 
hancements. 

Teachers shared their perspectives about the quality of the training sessions through teacher 
training feedback forms that were completed at the first and last sessions of each enhancement 
training. Teachers gave a high rating to the quality of training for all enhancements, mostly 
ranging from 4 to 5 on a scale of 1 (low) to 5 (high). On average, teachers rated the training ma- 
terial as accessible and easy to understand, and the trainers as supportive, interested, and con- 
cerned about teachers’ progress. Teachers also reported that the training helped to facilitate bet- 


^Wemer(2004). 
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ter relationships with the coaches and prepared them to successfully implement the enhance- 
ments’ strategies with their students, as this lead teacher in Tools of the Mind said: “The easiest 
part was having her [the coach] to help get it implemented. . .We got a good start with the Satur- 
day [training sessions].” 

Coaches also reported that the training sessions were of high quality and beneficial to 
teachers. Coaches rated the training highly: 4 to 5 on a scale of 1 (low) to 5 (high) on the coach 
training feedback forms. On average, coaches responded positively to questions that addressed, 
for instance, whether the material was presented in an accessible and easy-to-understand man- 
ner; whether suggested classroom activity assignments were useful; whether the trainer was 
supportive, interested, and concerned about the coach’s progress; and whether the coach would 
recommend the workshop to teachers. 

The research team attended the training sessions with an eye toward gaining a better 
understanding of how the enhancement content was being delivered. Based on qualitative ob- 
servations recorded in research team training observation forms, the researchers’ evaluations 
differed somewhat from the teacher training feedback forms, identifying more variation in how 
the content was delivered across training sessions. In other words, the research team identified 
some slightly lower-quality and higher-quality training sessions. The general conclusion from 
those evaluations, however, was that most training sessions were of satisfactory quality, in that 
they accomplished the intended goals of preparing teachers to implement the assigned en- 
hancement. 

Support by Trainers 

As part of each model, trainers also provided on-the-ground technical assistance to 
teachers and coaches in addition to conducting the training. In general, trainers for PATHS and 
Tools of the Mind were scheduled to visit classrooms once in the fall and once in the spring. 
The Incredible Years trainers usually stayed another day after training to either meet with 
coaches or observe classrooms, resulting in a greater number of trainer visit days in classrooms. 

• As planned, trainers visited classrooms between two and three times a 
year to support implementation of the enhancements. 

Trainer visits varied in terms of length, focus, and teachers’ perception of the visit’s 
usefulness. While some teachers thought the visits provided an important opportunity for them 
to check in with trainers and get hands-on support, other teachers found the visits less helpful. In 
general, teachers liked intensive, focused support from the trainers, and they believed that visits 
were not as helpful when they had less access to trainers. 


41 



Coaching in the Head Start CARES Demonstration 

Coaches played a critical role, ensuring that teachers understood the enhancement practices that 
they learned in the training and implemented them in the classrooms. The typical Head Start 
CARES coach was female, 46 years of age, and white. A typical coach had a graduate degree in 
early childhood education, a minimum of 1 1 years of experience in early childhood settings, 
zero to four years of experience in adult education, and minimal training as a coach in a social- 
emotional intervention. Coaches were either full- or part-time employees of the grantee, and 
they had not generally been part of the Head Start system before the demonstration began. Most 
coaches were hired at the start of the implementation year by the grantee. The technical assis- 
tance team provided support to grantees who wanted it during the hiring process. 

Fifty-two coaches supported implementation in the classroom; generally, each grantee 
hired three coaches, with each coach supporting a different enhancement. There was some turn- 
over in coaching staff over the course of each year of enhancement implementation. For Cohort 
One, three coaches were hired or started late after replacing coaches who left the demonstration. 
For Cohort Two, four coaches left in the course of the year. (A separate report provides more 
information about coach hiring and other aspects of the Head Start CARES coaching process.)^ 
This level of turnover was fairly low, and it did not seem to affect dosage and quality of the 
coaching component. 

Fidelity of coaching was assessed by examining the dosage and the quality of 
coaching that teachers received. The fidelity of the coaching component was measured by 
observing whether coaches gathered information about implementation of the enhancement 
from weekly hour-long classroom observations and conducted weekly 30-minute coaching 
meetings that reviewed teachers’ progress toward goals, addressed implementation-related 
issues, and made plans for future observations and meetings. Coaches were intended to be 
high quality, meaning that they were professional, steeped in early childhood development 
or teaching, and able to help teachers implement the enhancements well. Coaches were also 
required to meet with trainers to receive ongoing supervision. Consistent with the training 
component’s goal of creating a team of teachers immersed in the learning and application of 
the enhancement content, lead and assistant teachers were expected to attend coaching ses- 
sions together. 

Coaching Dosage 

As described above, coaches were expected to meet with lead and assistant teachers for 
30 minutes each week, in addition to conducting hour-long classroom observations weekly. 


^Lloyd and Modlin (2012). 
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Given teachers’ busy schedules and the additional demands of a comprehensive program such 
as Head Start — in which classrooms also provide children with meals, naps, and health initia- 
tives such as bmshing teeth — it was important to understand whether coaches were actually 
able to meet with teachers. 

• Dosage of coaching was closely aligned with what was prescribed. 

As shown in Table 3.3, coaches reported observing and meeting with teachers an aver- 
age of three times a month. (The goal was to meet four times.) Observations of teacher practice 
lasted over 70 minutes on average (60 minutes was the expectation) and meeting times across 
enhancements averaged 51 minutes, exceeding the expected 30-minute meeting time specified 
in the model. In total, teachers and coaches worked together, on average, about two hours per 
week. Although the coach weekly logs indicate that coaches met with teachers for longer than 
the prescribed time, qualitative and survey data from interviews with coaches as well as with 
lead and assistant teachers indicate that creating time for coach-teacher meetings was a fre- 
quent challenge. Teachers often had to meet a number of requirements during the school day, 
and it was hard to find 30 minutes when both adults could leave the classroom. Additional 
probing of coaches about these findings suggests that they may have been coaching teachers 


Head Start CARES Demonstration 
Table 3,3 

Number and Length of Coaching Meetings and Classroom Observations 


Coaching Components 

All 

Enhancements 

Incredible 

Years 

Preschool 

PATHS 

Tools of 
the Mind 

Coaching meeting 

Number throughout the year 

23.89 

23.69 

23.73 

24.25 

Number per month 

2.99 

2.96 

2.97 

3.03 

Average meeting length (minutes) 

51.02 

51.81 

54.17 

47.04 

Classroom observation 

Number throughout the year 

24.22 

24.34 

23.84 

24.49 

Average number per month 

3.03 

3.04 

2.98 

3.06 

Average observation length (minutes) 

77.25 

75.48 

73.95 

82.37 

Sample size” 

230 

77 

77 

76 


SOURCE: MDRC calculations based on the coach weekly logs. 

NOTES: Coach weekly logs were submitted weekly between September and April. 
“For all variables in the table, data are available for 100 percent of the sample. 
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infomially while the teachers were engaged in other classroom tasks. As discussed in more 
detail in Chapter 4 , teachers in part-day classrooms had especially limited time for coaching 
during the days when they provided services to double sessions of children, with one immedi- 
ately following the other.'* However, part-day programs often operated for only four days of 
the week, leaving the fifth day for teachers to complete their planning, handle other work, and 
potentially receive coaching. 

Coaching Quality 

In addition to the frequency of coaches’ meetings with teachers, fidelity to the coaching 
component required the coaches to deliver high-quality support. Enhancement trainers and 
teachers were the primary reporters of coaching quality. Trainers rated coaches using the trainer 
logs of coach quality an average of three times over the course of the implementation year on a 
scale of 1 (low) to 5 (high), which assessed items such as whether the coach ably demonstrated 
techniques or strategies of the enhancement; provided good feedback on observations using 
notes or data; helped teachers problem-solve about children, other staff, or center issues; and 
helped all teachers to implement the enhancement with fidelity to the model. 

• Coaches were of moderately high quality, based on trainers’ reports, 
and they felt effective in their positions. 

Trainers reported that the quality of the coaches was moderately high, rating them an 
average of 3.97 out of a possible 5. There was, however, some variation across the enhance- 
ments: as shown in Table 3.4, Incredible Years and Tools of the Mind trainers rated their coach- 
es at a moderately high level of quality,^ with averages of 3.74 and 3.89, respectively, and 
PATHS trainers gave their coaches strong ratings, with an average of 4.30. Generally, coaches 
were able to support the implementation of their enhancement, with more than 89 percent of 
them receiving a score of 3 or higher, although developers observed during a year-end summary 
review that coaches’ skill levels varied. Teachers also rated coaches highly in a lead teacher 
self-survey — an average of 4.63 for both the quality of coach support of implementation and of 
the coach-teacher relationship items (not shown in table). 


'*Full-day sessions lasted more than 3.5 hours. Part -day sessions lasted 3.5 hours or less, and some teachers 
taught two part-day sessions in one day. 

^Trainers rated coaches on a scale of 1 (low) to 5 (high) in the trainer logs of coach quality based on ob- 
servations of coaches and their classrooms, as well as trainers’ impressions of coaches from training sessions, 
meetings, and supervisory phone calls. Trainers responded to items such as “the coach is knowledgeable about 
the strategies or program and can answer teachers’ questions” and “the coach provides emotional support to his 
or her teachers.” 
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Head Start CARES Demonstration 
Table 3.4 


Trainers’ Ratings of Coach Quality, by Enhancement 


Enhancement 

Average Rating 
of Coach Quality 

Percentage 
of Coaches 
Rated at Least 3 

Percentage 
of Coaches 
Rated at Least 4 

N" 

Incredible Years 

3.74 

83.3 

38.9 

18 

Preschool PATHS 

4.30 

94.1 

76.5 

17 

Tools of the Mind 

3.89 

88.2 

52.9 

17 

All enhancements 

3.97 

88.5 

55.8 

52 


SOURCE: MDRC calculations based on the trainer logs of coach quality. 

NOTES: Trainers rated coaches 2-3 times on average over the course of the school year, 
between September and April. 

The ratings shown in the table measure 10 items and are assigned on a scale of 1 
(low) to 5 (high). 

^For all variables in the table, data are available for 100 percent of the sample. 


Trainers and teachers reported that successful coaches exhibited skill in three important 
areas: knowledge of the enhancement being coached, general coaching and consultation skills, 
and knowledge of and experience in early childhood development and/or teaching. 


Summary 

Overall, training and coaching in the Head Start CARES demonstration were implemented as 
intended. Although the quality of coaching varied somewhat across the enhancements, teachers 
generally received well-structured training and coaching on an ongoing basis, setting the stage 
for them to implement the enhancements well in the classrooms. The next chapter examines 
how classroom implementation of the enhancements actually unfolded during the Head Start 
CARES demonstration, given the amount and quality of professional development and support 
that was provided to teachers. 


45 



This page intentionally left blank. 



Chapter 4 

Implementation of the Enhancements in the Classrooms 


This chapter describes the extent to which the three Head Start CARES enhancements were de- 
livered in the classroom with fidelity to the original design. It provides an overview of what im- 
plementation looked like in the classroom for each enhancement and summarizes the successes 
and challenges of the implementation process. It also outlines organizational challenges and 
supports that appeared to be important to teachers’ classroom implementation. 

Overall, the Head Start CARES enhancements were implemented successfully: when 
rated on a scale of 1 (low) to 5 (high), they scored above the predetermined threshold of 3 (satis- 
factory). Coaches and trainers independently described classroom implementation as satisfacto- 
ry and continuously improving over the course of the year. When asked to describe their and 
children’s reactions to the enhancements in their classrooms, teachers and staff from the Head 
Start centers generally replied positively, reporting that they liked the enhancements, which they 
believed made a difference in children’s behavior. 

Implementation varied across enhancements, however; The Incredible Years and Pre- 
school PATHS were implemented with moderately high fidelity, and Tools of the Mind was 
implemented with lower but still generally satisfactory fidelity. Each enhancement had a unique 
set of activities that affected fidelity, but all the enhancements also faced some common chal- 
lenges, such as a lack of time, resources, and organizational support. The findings suggest that, 
in addition to training, coaching, and technical assistance, grantee support and resources helped 
teachers to implement the enhancements. 


How Was Fidelity Determined? 

Coaches used coach monthly fidelity logs to document their impressions of teachers’ implemen- 
tation of each enhancement’s core components. As explained in Box 4.1, these logs included 
sections in which coaches recorded their perceptions about areas of fidelity that were similar 
across the three social-emotional enhancements as well as those that were unique to each en- 
hancement. Across all fidelity instmments, items were scored on a scale of 1 (low) to 5 (high). 

Trainers completed trainer fidelity logs when they visited classrooms. These logs in- 
cluded a section in which trainers could record the extent to which both the lead and assistant 
teachers generalized and modeled the enhancement — for example, “Teachers are prepared for 
[enhancement] activities and seem familiar with what to do.’’ Trainers also provided fidelity 
ratings on the extent to which the teachers implemented enhancement-specific activities. 
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Box 4.1 

Maintaining Fideiity: Deiivering the Enhancements in the Ciassroom as 

Intended 

As with coaching and training, fidelity to the original design of the enhancements was meas- 
ured in terms of both quality and dosage (frequency and duration). High fidelity was defined 
as implementing each of the enhancement components exceptionally well and often, and low 
fidelity was defined as implementing each of the enhancement components poorly or rarely. 
Benchmarks that were predetermined by the technical assistance team were used to track and 
monitor fidelity. On a scale of 1 (low) to 5 (high), classrooms with a rating above a 3, mean- 
ing that teachers were implementing the enhancement occasionally but not consistently, were 
considered to be above the threshold for implementing satisfactorily. 

Fidelity was defined in two ways, taking into consideration coaches’ and trainers’ percep- 
tions about global components of fidelity that were similar across the three social-emotional 
enhancements and components that were unique to each enhancement. Global items focused 
on the general feel of the classroom and whether the enhancement appeared to be imple- 
mented well, based on responses to questions that sought to determine whether, for example, 
“it is clear when you enter this classroom and look around, it is a [specific enhancement] 
classroom” and “the children are actively engaged in [specific enhancement] throughout the 
day. It is not just seen as a special event.” Overall fidelity related to global components was 
determined based on an average of the responses to these questions. 

Enhancement-specific questions included items about components that were unique to each 
enhancement, such as weekly large-group lessons in Preschool PATHS and having the chil- 
dren draw and write “play plans” in Tools of the Mind. 

The full set of global and enhancement-specific items is listed in Appendix B. 


What Did Classroom Implementation Look Like? 

Given a sufficient level of support, including training, coaching, and technical assistance, it was 
hypothesized that teachers would be able to implement the enhancements well. The following 
sections detail what implementation of the enhancements looked like in the classrooms during 
the course of the implementation year and whether the enhancements were delivered in the 
classroom with sufficient fidelity to the original design. 

• All three enhancements were generally well received by teachers and 
Head Start staff. 
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Teachers reported that the enhancements made sense to them and that they were able to 
implement the enhancements, although they felt that some aspects were harder than others. In 
addition, they reported that implementing the enhancements made a difference in their class- 
rooms. Explaining how The Incredible Years has improved her ability to address children’s be- 
havioral issues in her classroom, one lead teacher said: 

A lot of times when children misbehave, you think, “My hands are tied. 1 can’t 
do time out, 1 can’t do this.” Instead of, “I can’t, 1 can’t, 1 can’t,” Incredible 
Years made me think, “1 can do this.” It has helped, it really has. It has helped 
me to come up with solutions with what I’m doing in my room. 

The perception among local Head Start staff, including lead and assistant teachers, cen- 
ter directors, and instmctional coordinators, was that the children benefited from the enhance- 
ments. A lead teacher who was implementing PATHS commented that the children not only 
learned how to control their own emotions but also learned to help other children control their 
emotions. She perceived that these lessons led not only to more emotionally well-regulated 
children, but also to improvement in how children interacted with each other: 

Just the behavior of the children, you know, there are the changes in them, the 
way that they speak to one another. . .more respect. If one of my children is up- 
set, they go to each other, “Do you need to do turtle?” or if one is upset: “You’re 
hurting my feelings and that makes me sad.” They pay good compliments to one 
another. 

An assistant teacher who was implementing Tools of the Mind also commented that 
Tools taught children to control themselves. She noticed that where children had floated be- 
tween play centers (play areas) during “center time” in an unregulated fashion before Tools was 
implemented, they now were able to play in one area for the full allotted time: 

1 think it helped on the self-regulation, too, because we in the past had the 
kids... go from center to center to center.... [T]his changed [in Tools] because 
they had to play for 30 minutes. And at first, we didn’t know how it was going to 
work, but. ..it helped; they finally got the hang of it. ..four- and five-year-olds 
got it. And they did regulate. 

• Fidelity of classroom implementation was above the predetermined 
threshold of “satisfactory.” 

As mentioned previously, coaches completed coach monthly fidelity logs, and trainers 
completed trainer fidelity logs, documenting the implementation process and rating fidelity. 
Trainers completed the logs when they visited classrooms,' typically two times over the course 


*For more information about the trainer visits, see Chapter 2. 
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of the year. The technical assistance team, in collaboration with the enhancement developers, 
set a predetermined threshold for assessing adequate global fidelity during the demonstration. A 
score of 3 on a scale of 1 (low) to 5 (high) was considered satisfactory, indicating that teachers 
were implementing the enhancement occasionally, though inconsistently. On average. Head 
Start CARES classrooms achieved, and often exceeded, this benchmark. Coaches gave class- 
rooms an average score of 3.68, while trainers scored classrooms a 3.25, averaging a 3.47 be- 
tween the two, as shown in Table 4.1. 


Head Start CARES Demonstration 
Table 4.1 

Coaches’ and Trainers’ Ratings of Fidelity of Classroom Implementation 


Perspective 


Average Rating 


All 

Enhancements 

Incredible 

Years 

Preschool 

PATHS 

Tools of 
the Mind 

Average coach and trainer perspective^ 

3.47 

3.69 

3.73 

2.97 

Trainer perspective 

3.25 

3.42 

3.63 

2.69 

Coach perspective 

3.68 

3.96 

3.82 

3.25 

Change over time (coach perspective)’’ 

0.75 

0.78 

0.72 

0.74 

Sample size’’ 

230 

77 

77 

76 


SOURCES: MDRC calculations based on the coach monthly fidelity logs and the trainer fidelity logs 
recorded in the management information system. 

NOTES: Coaches filed coach monthly fidelity logs each month, between September and April. 

Trainers rated classrooms an average of 2-3 times over the course of the school year, between 
September and April. 

The ratings shown in the table measure 10 items and are assigned on a scale of 1 (low) to 5 (high). 
““The average coach and trainer perspective rating is the average of the average coach perspective 
throughout the year and the average trainer perspective throughout the year. 

'’The change score is the difference between the coach rating from April or, if missing, March, and 
from September or, if missing, October. 

Tor all variables in the table, data are available for 100 percent of the sample, except for the 
“change over time” variable, for which data are available for at least 87 percent of the sample. 


Associations between coach fidelity and trainer fidelity ratings suggest that although 
coaches and trainers rated classrooms differently (with coaches generally assigning higher fidel- 
ity ratings than trainers did), coaches and trainers generally agreed about classroom implemen- 
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tation.^ The coaches’ higher ratings may be explained by their more frequent interaction with 
teachers, allowing them to have a more complete view of the teachers’ regular practices across 
many days. Trainers were less knowledgeable about teachers’ day-to-day practices and ob- 
served teachers for only one day per rating observation. However, although they were not as 
familiar with the classrooms, trainers had more expertise in the enhancement content and had 
more experience with implementation across many classrooms and grantees. The fact that the 
coaches’ and trainers’ ratings are associated so strongly suggests that the rating instmment did 
in fact measure the constructs it was intended to assess. 

• Fidelity of classroom implementation improved over the course of the 
school year for all enhancements. 

As shown in Figure 4.1, management information system (MIS) data (that is, the fideli- 
ty log ratings) from coaches show that over the course of the school year, from September to 
April,^ the average Head Start CARES classroom improved 0.75 point along a scale of 1 (low) 
to 5 (high). This is approximately three-fourths of a standard deviation of change, demonstrat- 
ing that over eight months, teachers were able to substantially and positively change their level 
of implementation. As shown in Figure 4.2, coaches reported that most Head Start CARES 
classrooms (83 percent) scored at least at the basic technical assistance threshold of 3 in Janu- 
ary, and in April, 60 percent of Head Start CARES classrooms scored at least a 4, indicating 
that they were implementing the enhancement well and consistently. 

Most (75 percent) of the teachers who started the year with low classroom implementa- 
tion scores ended the year at least at the threshold for satisfactory implementation (at or above a 
score of 3). Moreover, even teachers who began implementation well (at or above a 3) were 
able to improve: a majority of the teachers (64 percent) who started the year with a score be- 
tween 3 and 4 ended the year with a score at or above a 4. It is encouraging that teachers contin- 
ued to learn and improve classroom implementation over the course of the year; however, this 
change in implementation levels throughout the year also means that children did not necessari- 
ly receive high-quality implementation above the technical assistance threshold of a 3 for the 
whole year. 

• Implementation of the enhancement in the classroom with fidelity varied 
across enhancements, with moderately high levels for The Incredible 
Years and Preschool PATHS and a slightly lower level of fidelity for 
Tools of the Mind. 


moderately high correlation was seen between coach and trainer ratings, with a correlation coefficient 
(Pearson’s r) of 0.73. 

^The change score is the difference between the coach rating from April or, if missing, March, and from 
September or, if missing, October. 
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Head Start CARES Demonstration 
Figure 4.1 

Classroom Fidelity Scores: Change Over Time 



■ Start-of-year fidelity score 
□ End-of-year fidelity score 


All (N = 229) Incredible Years Preschool Tools of the 
(N = 77) PATHS Mind (N = 75) 

(N = 77) 


Enhancement 


SOURCE: MDRC calculations based on the eoach monthly fidelity logs. 

NOTE: Coaches filed coaeh monthly fidelity logs eaeh month between September and April. The “Start-of-year” 
score is from the coach rating in September or, if missing, Oetober. The “End-of-year” score is from the coach 
rating in April or, if missing, March. The “Change score,” shown above the eolumns, subtracts the start-of-year 
score from the end-of-year score. 

The scores shown in the figure reflect 10 items and are assigned on a seale of 1 (low) to 5 (high). 


Figure 4.2 

Distributiou of Classrooms, by Fidelity Scores 



Oetober (N = 202) January (N = 222) April (N = 211) 

Month 


H Score <3 
□ Score >3 and <4 
■ Score >4 


SOURCE: MDRC calculations based on the coach monthly fidelity logs. 

NOTE: The scores shown in the figure reflect 10 items and are assigned on a scale of 1 (low) to 5 (high). 
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As shown in Table 4.1, fidelity varied by enhancement. Coaches and trainers rated fi- 
delity as moderately high for The Incredible Years (3.69) and Preschool PATHS (3.73), while 
fidelity for Tools of the Mind implementation was not as strong (2.97), but still nearly reached 
the threshold of 3 that was considered to be satisfactory. Within these averages, individual class- 
rooms implemented their respective enhancements at different levels of fidelity. Figure 4.3 
demonstrates that, according to Incredible Years and PATHS coaches, about 70 percent of 
classrooms scored at least a 4 in April, while about 40 percent of Tools of the Mind classrooms 
had a similar score at the same time. Across all of the enhancements, however, there was still 
room for improvement in the level of classroom implementation. Although implementation lev- 
els differed across enhancements, the improvement in teacher implementation practices was 
similar across enhancements, ranging from a change in score of 0.72 point (PATHS) to 0.78 
point (Incredible Years). 


What Contributed to Classroom Implementation? 

This section explores the characteristics of the enhancements and the teachers that may have 
contributed to implementation, other determinants of implementation, and potential solutions to 
the challenges that arose. 

Characteristics of the Enhancements 

• The structure, goals, and activities that are intrinsic to each enhance- 
ment may have affected implementation in the classroom. 

Implementation and technical assistance data suggest that the variation across enhancements 
may be related to the sfructure of the enhancements themselves. It was easier for teachers to 
implement an enhancement that was scripted and that involved activities or skills with which 
they were familiar. For example, scripted language that was similar to what teachers were al- 
ready doing in classrooms made The Incredible Years easier to implement than Tools of the 
Mind. Likewise, Preschool PATHS gave teachers scripted support to implement relatively 
straightforward lessons in emotions and social skills. By contrast. Tools of the Mind required 
teachers to engage in more theoretically complex practices that were less familiar to them, and 
to change more of their daily practices, which may have been difficult to accomplish in one year 
of implementation. 
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Head Start CARES Demonstration 
Figure 4.3 

Distribution of Classrooms, by Fidelity Scores, for The Incredible Years, 
Preschool PATHS, and Tools of the Mind 



October (N = 65) January (N = 77) April (N = 75) 



October (N = 66) January (N = 69) April (N = 64) 



■ Score >4 □ Score >3 and <4 S Score <3 


SOURCE: MDRC calculations based on the coach monthly fidelity logs. 

NOTE: The scores shown in the figure reflect 10 items and are assigned on a scale of 1 (low) to 5 (high). 
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The Incredible Years 

The content of The Incredible Years Teacher Training Program focused heavily on 
building positive relationships with children and their families and teaching proactive classroom 
management techniques. The Incredible Years enhancement is based on best practices for early 
childhood teaching with minimal focus on preparing teachers to present specific lessons or con- 
tent. On the one hand, this approach made it easier for teachers to implement the Incredible 
Years enhancement, as they were generally familiar with many of the practices being imple- 
mented and they did not need to alter their classroom schedule to allow for additional lessons 
and activities. Many teachers even commented that the Incredible Years practices were similar 
to mles and practices that they were already using. As one lead teacher observed, “A lot of the 
same mles that we learned in Incredible Years are very close to the classroom mles we have.” 

On the other hand, some teachers needed to adapt to the “Incredible Years way” by 
changing preexisting classroom management techniques. For example, faithful Incredible Years 
implementation required that some teachers modify their moment-to-moment interactions with 
the children in order to establish new classroom management norms. One lead teacher recalled 
stmggling at the beginning of the year with the practice of ignoring the conduct of children who 
were misbehaving, a sentiment that other teachers expressed as well: “Ignoring was very diffi- 
cult. If there was a behavior that a child wanted to get attention for, it was really hard to just let 
the child keep doing it.” 

Teachers were, however, used to some of the more common behavior management 
techniques, such as praising or complimenting. As shown in Table 4.2, fidelity was high for 
practices that are specific to The Incredible Years, but the level of fidelity was lower for practic- 
es that were less familiar to teachers or different from general practices that teachers were al- 
ready using in the classroom. For example. The Incredible Years has teachers create behavior 
plans for children who are being dismptive; these plans lay out specific consequences for indi- 
vidual behaviors that a child is exhibiting. This more intensive behavior management practice, 
unique to the Incredible Years enhancement, was scored lower (3.5 out of 5) on fidelity than 
other components, such as commenting on good behavior (4.05), an activity with which most 
early childhood educators are already familiar. 

Preschool PATHS 

PATHS lessons and extension activities take up a clearly defined and limited amount of 
time each week, with one lesson and one extension activity lasting about 30 to 40 minutes for 
the week. The program also includes a regular daily activity (called “Kid of the Day,” in which 
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Head Start CARES Demonstration 
Table 4.2 

Fidelity Scores on Specific Components of 
The Incredible Years in the Classroom 


Component 

Score“ 

Comments on good behavior 

4.05 

Clear classroom rules 

4.01 

Ignoring'’ 

3.85 

Behavior plans 

3.50 

Sample size*^ 

77 


SOURCE: MDRC calculations based on the Incredible Years 
coach monthly fidelity logs. 

NOTES: Coaches filed coach monthly fidelity logs each 
month between September and April. 

“The scores shown in the table measure 13 items and are 
assigned on a scale of 1 (low) to 5 (high). 

'’Data for “Ignoring” start in February. 

‘’For all variables in the table, data are available for 100 
percent of the sample. 


one child is given a special job and receives compliments from the teachers and the other chil- 
dren) and “generalization” (in which teachers integrate the themes and core practices of the en- 
hancement into various activities throughout the day). As shown in Table 4.3, the more concrete 
aspects of PATHS were more easily implemented, with PATHS lessons, which are highly 
scripted, rating an average fidelity score of 3.76 over the course of the year, while fostering self- 
regulation, which was a later and less concrete part of PATHS, averaged 3.08. Many teachers 
explicitly pointed out PATHS’ scripting as a reason for its ease of implementation, as this lead 
teacher remarked: 

1 think because it is just scripted, you can read it, you can either follow the script, 
or you can put your own words into that situation because it’s similar. . .1 think 
that if you are able to kind of go along with the script, it’s easy. 

This relatively contained, highly scripted enhancement was easy for teachers to access 
and prepare, even though they hadn’t implemented it before. 
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Head Start CARES Demonstration 


Table 4,3 

Fidelity Scores on Specific Components of 
Preschool PATHS in the Classroom 


Component 

Score” 

Building a caring classroom: structure 

3.79 

PATHS lessons 

3.76 

“Kid of the Day” and compliments 

3.76 

Building a caring classroom: relationships 

3.76 

Nurturing emotional understanding 

3.56 

Fostering self-regulation 

3.08 

Sample size*’ 

77 


SOURCE: MDRC calculations based on the Preschool PATHS 
coach monthly fidelity logs. 

NOTES: Coaches filed coach monthly fidelity logs each month 
between September and April. 

““The scores shown in the table measure 10 items and are assigned 
on a scale of 1 (low) to 5 (high). 

'’For all variables in the table, data are available for 100 percent 
of the sample. 


Tools of the Mind 

As previously described in Chapter 2 , the content in Tools of the Mind was condensed 
from a comprehensive curriculum as part of a two-year implementation process to an “en- 
hancemenf’ that could be implemented over the course of one year. Tools of the Mind was, 
however, tested under the same conditions as the other enhancements, with teachers randomly 
assigned to the enhancement and supported by a strong training and coaching professional de- 
velopment model. 

Even in abbreviated form. Tools of the Mind required teachers to rearrange their class- 
rooms or toys when a new play theme was started and to implement a set of new or less com- 
mon activities daily. Some of the activities (buddy reading, message of the day, fmgerplays)'' 
were supported with well-prepared manuals and other classroom materials, and, according to 
teachers, were easy to implement. As one lead teacher explained: 


"^“Buddy reading” is when children take turns reading to each other. “Message of the day” is an activity in 
which the teacher and the children work to write a message. “Fingerplay” is any game that is acted out with the 
fingers, like “This Little Piggy.” 
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The must dos, those were simple. . .1 always had a message of the day; it proba- 
bly wasn’t as detailed as how Tools desires, but 1 would definitely write with 
children every morning. 

Other activities, such as play planning and play time, which required substantial scaf- 
folding and support of children by both the lead and assistant teachers,^ demanded a significant 
amount of teacher effort. These activities were theoretically complex: in addition to implement- 
ing the activity, teachers had to make knowledgeable decisions in the moment about how to 
support children and children’s play, even if they had just learned the theory. For example, even 
by the end of the year, this lead teacher said the class still struggled with play plans: 

To me, [play plans are] still a little difficult; it’s still a little shaky... Some of 
them that are artists or drawers, you know, it [the plan] would look more ad- 
vanced, [because they] would draw their plans and move [on],... whereas with 
some of them, that concept just didn’t register. It would take longer then, the 
concentration would stop, they’re looking around and then they would just 
scribble and give something. That was hard. 

Teachers were asked to actively engage with children in make-believe play and to scaf- 
fold children’s efforts when they were playing with each other. This approach often contrasted 
with previous teacher behaviors during play time, when they typically focused on completing 
paperwork or relaxing and playing with the children.'’ As shown in Table 4.4, teachers did a bet- 
ter job of implementing “graphics practice^uddy reading,” which received an average fidelity 
score of 3.26 over the course of the year, while the more complex and time-intensive make- 
believe play practice, which was a key activity of the Tools enhancement, received a score of 
only 2.62. 

Characteristics of the Teachers 

• Some teacher characteristics, such as age and motivation to imple- 
ment the enhancement, were associated with a teacher’s implementa- 
tion fidelity. 

In addition to the characteristics of the enhancements, it was possible that teacher char- 
acteristics, as shown in Table 4.5, were also associated with fidelity to the design of the en- 
hancement as delivered in the classroom. To explore this issue, analyses were conducted of ass- 


^Recall from Chapter 2 that scaffolding is a practice that teachers use to support a child’s activity or re- 
sponse at his or her current level of understanding while extending the activity or response in order to push the 
child to the next level of ability. 

^Other studies have shown that teachers spend as much as 60 percent of free play time completing tasks 
such as talking with other teachers and parents and handling administrative duties. For more information, see 
Coplan and Prakesh (2003); Kontos (1999). 
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Head Start CARES Demonstration 


Table 4,4 

Fidelity Scores on Specific Components of 
Tools of the Mind in the Classroom 


Components 

Score” 

Graphics practice/buddy reading 

3.26 

Play planning and scaffolded writing 

3.10 

Make-believe play 

2.96 

Make-believe play practice 

2.62 

Sample size*’ 

76 


SOURCE: MDRC calculations based on the Tools of the Mind 
coach monthly fidelity logs. 

NOTES: Coaches filed coach monthly fidelity logs each 
month between September and April. 

^The scores shown in the table measure 6 items and are 
assigned on a scale of 1 (low) to 5 (high). 

’’For all variables in the table, data are available for 100 
percent of the sample. 


ociations between lead teachers’ baseline characteristics, collected via a teacher self-survey (at 
baseline), and scores reflecting fidelity of classroom implementation. In the time between base- 
line in the spring and implementation during the school year, some teachers left the study and 
were replaced by new teachers. Therefore, some teachers do not have baseline data available 
and are not included in these analyses. 

The results show that age was somewhat negatively associated with the teacher’s fideli- 
ty.’ That is, older teachers implemented the enhancements less faithfully than did younger 
teachers, a similar pattern to that seen in previous research.* Although there is no definitive evi- 
dence to explain this finding, it may be that younger teachers are more open to taking advantage 
of new practices than older teachers are. Somewhat surprisingly, teachers’ level of experience, 
years teaching, and years of education were not associated with teachers’ implementation of the 
enhancements, although, as can be seen in Table 4.5, there was little variability in some of 
these, particularly teachers’ experience level. Taken together, these findings suggest that there is 


’The correlation and p-value are as follows: r = -0. 15 ; p = 0.066. 
*Klimes-Dougan et al. (2009). 
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Head Start CARES Demonstration 


Table 4,5 


Selected Baseline Characteristics of Lead Teachers 
in the Program Sample 


Characteristic 

Mean 

Standard Deviation 

Age (years) 

44.54 

11.87 

Education (%) 

Less than an associate’s degree 

8.82 

- 

Associate’s degree but no bachelor’s degree 

28.82 

- 

Bachelor’s degree but no graduate degree 

55.29 

- 

Graduate degree 

7.06 

- 

Teaching experience level (%) 

Novice (0 to less than 3 years) 

5.33 

- 

Experienced (3 to less than 10 years) 

26.04 

- 

Seasoned (10 years or more) 

68.64 

- 

Years teaching 

16.26 

9.87 

Scored measures** 

Motivation to implement (1-5) 

4.18 

0.77 

Burnout (0-54) 

13.16 

10.54 

Reactions to childrens’ negative emotions (0-4) 

3.55 

0.56 

Lead teacher-teaching assistant relationship (1-5) 

4.51 

0.80 

Adaptability (0-4) 

3.07 

0.62 

Sample size** 


171 


SOURCE: MDRC calculations based on the spring teacher self-survey (baseline). 

NOTE: “The scale (range of seores) for each measure is shown in parentheses. 

’’For all variables in the table except for “Motivation to implement,” the 
sample size is 171, and data are available for at least 94 percent of that sample. 
Data for “Motivation to implemenf’ are available for 184 teachers beeause they 
were eollected at a later date, after some teachers in the sample had already left 
and were replaeed. 


no clear relationship between age, credentials, or experience and implementation of the Head 
Start CARES enhancements, and in fact teachers of varying levels of experience and back- 
ground are likely equally able to implement them successfully. 

Teachers’ implementation of the enhancements with fidelity was also positively associ- 
ated with their motivation to implement the intervention at the beginning of the school year.'* At 


’The correlation and p-value are as follows: r = 0.23; p = 0.002. 
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the beginning of the year, teachers were asked whether they felt motivated to use the strategies 
in their classrooms. Teachers who reported higher levels of motivation to implement the en- 
hancement in their classrooms at the start of the school year also showed higher levels of fideli- 
ty to the original model when delivering the enhancement in the classroom. This finding aligns 
with recent work demonstrating that practitioners’ pre-implementation beliefs and enthusiasm 
about an enhancement are positively associated with the number of intervention sessions deliv- 
ered, a measure of intervention fidelity. While not surprising, so little is known about what 
leads teachers to implement evidence-based practices that this connection between teachers’ 
self-described motivation to implement an intervention and their actual implementation of the 
enhancement adds important information to the field’s preliminary knowledge base. 

Delivering the enhancement as intended in the classroom was not significantly associat- 
ed, however, with teachers’ “bumouf ’ at the beginning of the study, teachers’ reactions to chil- 
dren’s negative emotions, the relationship between the lead and assistant teachers as rated by 
lead teachers, or the teachers’ view of their own adaptability to change. Notably, in this demon- 
stration, there seems to be little variability in many of these more nuanced teacher psychosocial 
measures. This low variability may have contributed to the lack of associations between teacher 
characteristics and fidelity. 

Other Determinants of and Barriers to Implementation 

Aside from enhancement and teacher characteristics, other factors such as resources, 
time, and language played a role in how well teachers could implement the enhancements. This 
section explores some of those factors and the solutions derived to address them. 

• Significant challenges to classroom implementation arose, but teachers, 
coaches, and grantees developed creative solutions for many of them. 

Although implementation of the enhancements was satisfactory or better, teachers and 
coaches identified barriers to implementation throughout the year. Not all of the issues affected 
teachers equally across grantees and enhancements, however. 

In general, barriers to implementation included (1) a lack of resources and materials; (2) 
insufficient time for teachers to plan and meet with coaches; (3) a mismatch in language among 
children, teachers, and coaches; (4) the absence of support for teachers and coaches from the 
Head Start centers; (5) additional curricular or assessment requirements; and (6) Head Start per- 
formance standards monitoring. Teacher turnover and teacher stress also made classroom im- 


'°Klimes-Dougan et al. (2009). 
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plementation more difficult. Despite these difficulties, teachers, coaches, and grantees often 
worked together to develop creative and context-appropriate solutions. 

Resources and Materials 

A few of the classrooms lacked basic supplies like materials, books, and toys that were 
needed to support enhancement implementation. It became challenging for those low-resource 
classrooms to implement certain activities — for instance, when they were unable to make pho- 
tocopies of turtle pictures in Preschool PATHS classrooms or did not have enough books to 
support buddy reading in Tools of the Mind classrooms. Coaches and teachers helped support 
the need for basic supplies by making toys from materials found at home or in low-cost or 
secondhand stores. To proactively address these issues in Cohort Two centers, teachers were 
provided with a $100 gift card to an educational store to buy supplies as needed for implementa- 
tion. Teachers and coaches worked together to identify the class’s implementation needs and to 
make decisions about how best to spend the gift card. 

Time to Plan and Meet with Coaches 

Teachers and coaches consistently described challenges in finding time to meet with 
one another, despite data showing that coaching meetings happened frequently and for longer 
than expected. One way grantees supported teacher-coach meetings was to hire “floating” sub- 
stitutes for the days when the coach visited the center, so that teachers could rotate out of the 
classroom for their coach meeting without being concerned that their classrooms would lack an 
educator. Coaches also reported “coaching on the fly,” meaning they worked and reflected with 
teachers throughout the day in the classroom, instead of in a separate, more formal meeting. 

Teachers in Tools of the Mind classrooms may have been even more pressed for time 
than teachers implementing Incredible Years or Preschool PATHS, because Tools required a 
substantial amount of time to implement all of the activities and intensive amounts of the teach- 
ers’ focus and attention on each individual child to scaffold their play and learning. In addition, 
as described previously, the Tools enhancement included changes in learning themes through- 
out the year, requiring teachers to spend time (and sometimes money) to create new props and 
items for implementing the theme. 

Teachers in double-session, part-day classrooms also had difficulty finding time to im- 
plement and plan. In Head Start CARES, some of the part-day classrooms operated as double 
sessions, where teachers taught one class in the morning and a second, separate class in the af- 
ternoon. To accommodate teachers’ need to plan and make home visits (which are part of the 
Head Start system of services to families), most part-day programs also offered programming 
for four days a week instead of five days. The classroom time for each session was often very 
short (3.5 hours or less), and teachers had two groups of children to plan for with little time dur- 
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ing the day to prepare. Coaches and teachers in such part-day classrooms identified the lack of 
time as a barrier to implementation, as described by this lead teacher in PATHS: “I work in a 
half-day program. The morning session has breakfast and lunch, and mandated play time. So, it 
can be very difficult to get everything in.” 

Interestingly, fidelity to the design of the enhancement among teachers in full-day class- 
rooms (with a mean score of 3.41 out of 5) was actually significantly lower than fidelity among 
part-day classrooms (with a mean score of 3.74 out of 5)." In other words, despite the lack of 
time, coaches and trainers reported that the enhancements were more likely to be implemented 
as intended in part-day classrooms than in full-day classrooms. Although the reason for this 
finding is unclear, there are a few possibilities. Perhaps the amount of time in part-day class- 
rooms that is devoted to instmction and active participation is higher compared with that in full- 
day classrooms, which also include time for naps, lunch, and other, less formal activities. Less 
“down time” in classrooms may force teachers to be more diligent about ensuring that imple- 
mentation occurs throughout the day. Alternatively, part-day classroom settings may have made 
teachers, coaches, and centers more committed to overcoming these time challenges. As noted 
above, part-day programs generally operated four days a week, and teachers had one day avail- 
able for planning and other responsibilities. Coaches and teachers may have been more cogni- 
zant of the lack of time and had more focused or more structured coaching sessions or taken 
advantage of the one day a week that teachers had free for planning. 

Language Mismatch 

While teachers were trained in the enhancements in English with the expectation that 
they would also implement the intervention in English, that did not always happen. Some teach- 
ers who spoke English as a second language stmggled during the training sessions because the 
content was delivered in English. Teachers in bilingual classrooms often delivered the en- 
hancement to children in Spanish, the children’s language of comfort, in an effort to better en- 
gage them in the activities. Because the enhancement materials were not available in Spanish, 
translations may have been less than accurate, as teachers often translated the material in the 
classroom as they were talking to children. In some cases, only one member of the teaching 
team spoke Spanish, making implementation on the part of both teachers a greater challenge. 
Additionally, often the coaches were not hired to match the linguistic make-up of the Head Start 
centers. This made coaching difficult, as coaches were not able to understand non-English inter- 


*’The difference in the fidelity scores between full- and part-day classrooms was statistically significant at 
the p = 0.01 level. That is, there was a 1 percent probability that the difference was a result of chance alone, 
meaning that the difference could be attributed to the type of classroom with a high degree of confidence. 
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actions among children or between teachers and children. In those situations, coaches had to 
work with both teachers to come up with solutions; for instance, the teachers might translate for 
the coach as she watched or the assistant teacher might conduct some lessons in Spanish for 
children who needed it. 

Level of Support from the Head Start Center 

Some teachers were also affected by other center-level factors, such as lack of support 
or a difficult space in which to work. Cross-classroom support and collaboration within centers 
was limited; teachers basically operated on their own in the classrooms. Lead and assistant 
teacher interviews revealed that they valued whatever opportunities they had to communicate 
with other teachers. Oftentimes, these opportunities occurred during training, according to this 
Tools of the Mind lead teacher: 

1 love gleaning from other educators, so 1 enjoyed the initial training because 
more people were there. Even if just another school was there, we could compare 
and see how things are working for them. 1 like having a fresh set of eyes and 
fresh perspective. It might motivate you to look at things a little differently. 

In addition to interpersonal support, some teachers also needed support to manage the 
physical environment in the center. In a few cases, teachers had to pack up their classrooms at 
the end of the week so the room could be used for other purposes. The need to pack up their 
belongings may have cut into teachers’ time for implementation or coaching; in particular. 
Tools teachers in this situation sfruggled because they needed to continuously recreate elaborate 
make-believe play areas. 

Additional Curricular or Assessment Requirements 

Teachers were challenged to implement multiple curricula, special programs, and child 
assessments at the same time that they were implementing their Head Start CARES enhance- 
ment. All 1 7 grantees were already using a base curriculum — usually Creative Curriculum or 
High/Scope — and implementing the Head Start CARES enhancements on top of it. To bal- 
ance the competing implementation needs of these curricula or assessments along with the 
Head Start CARES enhancements, developers created “crosswalks” to help teachers and pro- 
gram administrators understand how each enhancement fulfilled other program requirements. 
For example, developers might point out that documentation of child interactions during a spe- 
cific social-emotional activity in their enhancement would also help teachers fulfill the re- 
quirement for documenting social-emotional school readiness. While these crosswalks were an 
important first step, it was still often difficult for teachers to balance competing demands for 
their time and attention. 
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National Head Start Performance Standards and Administrative Monitoring and 

Oversight 

In addition to ongoing classroom demands and requirements, all Head Start centers are 
subject to national Head Start performance regulations. These requirements are accompanied by 
a strict and fixed monitoring schedule, a high level of administrative oversight, and significant 
consequences for poor performance. 

For this reason, adhering to the requirements of the Head Start CARES project was not 
a simple undertaking for grantees that were scheduled for a monitoring visit during the imple- 
mentation year. In some cases, particular aspects of the enhancements created real or perceived 
conflicts with Head Start regulations, leading to a heightened sensitivity on the part of grantee 
administrators and classroom teachers about implementing activities that monitors might view 
as outside of usual and required practice. The need to prepare for monitoring visits also created 
additional demands on teaching staff As a counterpoint, some grantees reported that implemen- 
tation of the enhancements strengthened their classroom programs and contributed to perform- 
ing well on the monitoring requirements. 


Which Organizational Characteristics Are Important for 
Implementation? 

Organizational characteristics and support appeared to be integral to whether teachers could im- 
plement the enhancements well in their classrooms. Organizational support manifested itself in 
many ways, including the communication of support for the demonstration and implementation 
of each enhancement, the provision of materials, and how well grantees thought the enhance- 
ments fit with their mission. 

The design of the research study was also a factor. Grantees were asked to allow their 
centers to implement three separate enhancements at the same time, while being encouraged to 
remain neutral and not favor a particular enhancement over another. In an effort to remain neu- 
tral, administrators may have been effectively silent about all the enhancements. 

The Head Start CARES data do not formally reflect the relationship between organiza- 
tional capacity and fidelity. However, organizational support in the demonstration was evaluat- 
ed from many different perspectives. Coaches, trainers, and developers completed written as- 
sessments based on their interactions with and knowledge about grantees throughout the year. 
The research team assessed the grantees based on site visits and the technical assistance they 
provided. In order to identify characteristics of a strong grantee, coach assessments, trainer as- 
sessments, developer assessments, research team assessments, and a qualitative ranking were 
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combined into a single rating system.'^ The ratings of grantees that the research team gathered 
from site visits suggest that organizational capacity was a key facilitator of quality professional 
development and classroom implementation of the enhancements. 

• Grantees that provided higher levels of organizational support did so in 
different ways, including sending encouraging messages about imple- 
mentation, hiring appropriate staff in a timely manner, providing re- 
sources for implementation, and helping teachers overcome obstacles 
such as a lack of time. 

Grantees that were better able to support implementation sent strong messages of sup- 
port for Head Start CARES implementation to their teachers, hired and provided administrative 
supervision for coaches, helped efforts to crosswalk and integrate the enhancement with their 
core curriculum, made time and space available for teacher training, and devised solutions to 
time management challenges, such as providing “floating” teachers or substitutes during coach- 
teacher meetings, as described earlier. 

Grantees with a high level of organizational support had backing from multiple mem- 
bers of the administration. Center directors or administrators may not have unanimously sup- 
ported or been involved in the implementation process, but more than one staff member with 
administrative clout tended to work closely with the teachers and coaches to actively support 
enhancement implementation. In some cases, the center administrator attended training sessions 
and sometimes even drove teachers to training themselves. In other cases, the enthusiastic sup- 
port came from mental health specialists, education coordinators, or education specialists.'^ In 
highly supportive grantees, additional staff members or administrators who were neither grantee 
liaisons nor teachers were consistently and actively involved in implementation. 

In many cases, these grantees also established concrete performance expectations for 
teachers. The coach and grantee liaison communicated regularly with the center administrators, 
educational specialists, and other center-based staff The grantee liaison and administration 
showed interest in what was happening in the classrooms and checked in to make sure that im- 
plementation was continuing as expected. In some centers, this came in the form of regular 

*^At the end of implementation, but before any impaet findings were known, MDRC researeh and site 
teams held a final meeting to reflect on implementation using information collected from a range of MIS and 
interview data, as well as data from coach calls, developer meetings, and other sources. The team rated each 
grantee, by enhancement, on the quality of training, coaching, classroom implementation, and organizational 
support. The ratings were assigned on a scale of low, medium- low, medium, medium-high, and high, and were 
reached through a process of consensus building. 

'^Head Start provides comprehensive education, physical health, and mental health services to low-income 
children. Education coordinators and specialists help support curriculum and educational programs, and mental 
health specialists work with classrooms and individual children with regard to mental health services. 
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meetings; in other centers, administrators or education specialists made themselves available 
for questions. 

Conversely, it also seemed important that administrators were flexible and not too mi- 
nutely involved in implementation. Some teachers and coaches commented that strong adminis- 
trators were “hands off’ and did not “micro-manage,” which made teachers feel supported and 
tmsted in their work. Teachers’ personal time was respected and teachers were able to change 
their work schedule to suit their needs and adapt it as they went through training. In grantees 
with strong organizational support, teachers were permitted to drop some of the responsibilities 
that they previously had in order to implement the enhancement. Building on a more complete 
understanding of the crosswalks, administrators were able to understand why certain lesson 
plans may have been replaced or became more relaxed about curricula that weren’t specific to 
Head Start CARES. 

Finally, grantees that were characterized as offering strong organizational support tend- 
ed to provide teachers and coaches with all the materials that they needed. Materials were usual- 
ly easy to obtain, and teachers were reimbursed relatively quickly, if necessary. In most cases, 
materials were provided and teachers did not have to lay out any money. These grantees were 
characterized as being generally positive and maintaining supportive environments. Staff from 
more than one such grantee said they felt that they were part of a family. 

• Among grantees who were ranked lower on organizational support, 
messaging and communication between center and grantee staff were 
consistently cited as challenges. 

In some cases, grantee- or center-level administrators or directors sent mixed messages 
with regard to support for Head Start CARES implementation. They might, for instance, re- 
quire teachers to focus on other center-level issues that diverted time and attention away from 
Head Start CARES implementation. Some grantees were implementing multiple interventions 
during the school year, without providing follow-up to help teachers prioritize the demands on 
their time. At other times, teachers described feeling disconnected from grantee staff and any 
message of support. In part, this may have been the result of turnover in key management posi- 
tions and management structures in which center staff had limited interaction with upper-level 
administration. 

• Grantee perspectives on specific enhancements may have influenced 
implementation in the classroom. 

Throughout the course of implementation, most Head Start CARES grantees developed 
opinions about the fit and suitability of the enhancements for their centers despite the research 
requirement that grantees remain neutral on this score. A grantee’s negative perception of an 
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enhancement may have affected organizational support and had a negative effect on teachers’ 
motivation or implementation with fidelity to the original enhancement design. Although teach- 
ers’ and administrators’ perceptions of a particular enhancement tended to be similar within a 
grantee, each of the three enhancements was perceived differently within grantees; for example, 
a grantee might find that one enhancement was more consistent with its approach to providing 
services for children than another enhancement. Concerns about a specific enhancement gener- 
ally focused on how it fit with the classroom curricula. Head Start performance standards, or 
program philosophy. 

In general. Incredible Years and PATHS teachers and coaches felt supported by grant- 
ees, while Tools of the Mind teachers and coaches felt less supported. This difference may be 
related in part to Tools’ complexity and origins as a full curriculum, as well as what some grant- 
ee liaisons saw as a “prescribed” or teacher-directed approach to classroom learning. As ex- 
plained earlier. Tools of the Mind involved more changes to the classroom stmcture, therefore 
requiring increased flexibility and comprehension on the part of the administrators to ensure that 
other criteria were still met. Several Tools trainers speculated that a lack of grantee support af- 
fected implementation because, as one trainer said, “the teachers seemed particularly concerned 
with their jobs and did not want to do anything they felt the supervisor didn’t support” or be- 
cause, according to another, “the teachers just don’t really want to do [the enhancement]” with- 
out the administration’s ongoing support and presence. Poor organizational support for The In- 
credible Years tended to focus on issues of Head Start performance requirements, which grantees 
sometimes perceived as conflicting with the enhancement. The research team did not learn of 
any instances in which PATHS conflicted with the organization’s mission or vision. 

Overcoming such resistance was possible, however, and some grantees supported im- 
plementation despite concerns about some aspect of the Head Start CARES enhancements. In 
one case, the grantee made minor concessions on agency policies and practice to support im- 
plementation despite some discomfort with two enhancements. This may have made a differ- 
ence in the teachers’ attitudes because it communicated that the agency cared how well and 
completely the enhancements were implemented by showing their interest in minimizing the 
challenges in the classroom. 


Summary 

Fidelity of classroom implementation exceeded the predetermined threshold of “satisfactory” 
and improved over the course of the year. Fidelity varied across enhancements, possibly based 
on the stmcture and content of each enhancement. Teachers and coaches worked together to 
find creative solutions to implementation challenges. Organizational support and resources, as 
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well as teachers’ characteristics, influenced the quality of implementation. Teachers who started 
with greater motivation to deliver the enhancements and younger teachers were slightly better at 
implementation than were less motivated and older teachers, on average. When grantees com- 
municated openly, were receptive to the enhancement, and provided teachers and coaches with 
the time and materials necessary to implement the intervention, classroom implementation went 
more smoothly. 
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Chapter 5 

Changes in Teacher Practice 


This chapter explores whether implementation of the three Head Start CARES program en- 
hancements changed teachers’ day-to-day practices in the classroom. It presents the first im- 
pacts (or effects) of the enhancements by comparing teacher practices in the Head Start centers 
where the enhancements were implemented with teacher practices in the control group (or 
“business as usual”) centers. 


Did Implementation of the Enhancements Lead to Changes in 
Teacher Practice? 

According to the theory of change for Head Start CARES, faithful implementation of the inter- 
vention should lead to positive change in teacher practice. Previous research across a range of 
social service interventions has found that the extent to which an intervention was implemented 
with fidelity to its original design was strongly associated with the intervention’s effect on out- 
comes of interest. ' In the Head Start CARES demonstration, changes in teacher practice were 
conceptualized as “first order” effects; that is, teacher practice was considered to be the direct 
target of these enhancements. It was hypothesized that in order to see change in “second order” 
outcomes such as classroom interactions or children’s social-emotional behaviors, change 
would first need to be observed in the “first order” practices. 

Findings in this report demonstrate that implementation of the enhancements in the 
classroom was satisfactory or better. Given that coaching and training were implemented well 
and implementation of the enhancements in the classroom was described as satisfactory or bet- 
ter, the next logical question was whether this level of implementation was sufficient to produce 
measurable changes in teacher practice. 

• Each enhancement emphasized a specific teacher behavior that was dis- 
tinct from those emphasized in the other social-emotional enhancements. 

Given the different focus of each enhancement, it was hypothesized that each would 
emphasize a different teacher skill or practice in the classroom. For instance. Tools of the Mind 
— Play focuses directly and explicitly on teachers’ scaffolding of children’s behavior during 
play — that is, bolstering children’s current ability to play and supporting them in their progress 
toward a more complex level of play. The other enhancements did not as directly emphasize 


*Martinez-Beck (2013). 


71 



scaffolding of children’s “pretend play” behavior. In fact, each enhancement emphasized a par- 
ticular practice that would be expected to change once teachers implemented that enhancement. 

However, the specific set of skills and behaviors that each enhancement directly em- 
phasize were not necessarily the only teacher practices that the enhancement could have influ- 
enced. Each enhancement addressed a number of different teacher behaviors, including some of 
the primary teacher practices of the other enhancements. For instance. The Incredible Years fo- 
cused on classroom management, but also trained teachers to help the children with social prob- 
lem-solving and emotion regulation. Preschool PATHS targeted emotional competence, self- 
regulation, and social problem-solving development in children, but also promoted good class- 
room management as an integral part of a teacher’s ability to instmct children on the other 
skills. Tools of the Mind emphasized the support of teachers’ scaffolding behavior, but also at- 
tempted to change teachers’ use of time for transitions between activities, teachers’ interaction 
with children’s play, and teachers’ behavior and practices during literacy activities, among 
many others. A content-focused review of the three enhancements’ lessons or manuals that was 
conducted before Head Start CARES began demonstrated that the enhancements share some 
content, but the emphasis of each enhancement is different.^ 

The teacher skills and practices that each enhancement targeted directly could also be 
observable to some extent in classrooms where the intervention was not delivered. That is, 
teachers in the control group may have used classroom management techniques, instmcted chil- 
dren in social-emotional skills, and scaffolded children’s interactions. 

Given this possibility, it was hypothesized that: 

1. The Incredible Years would demonstrate an impact on teachers’ class- 
room management skills. Strengthening classroom management skills and 
building positive relationships with students and their parents are main targets 
of The Incredible Years and include the teachers’ use of a consistent routine; 
preparedness for classroom activities; awareness of what is happening in the 
classroom at all times; use of persistence, social and emotional coaching strat- 
egies, and proactive behavior management techniques, such as praising and 
rewarding good behavior and providing clear consequences; minimal use of 
negative behavior techniques, such as yelling or harshness; and use of ges- 
tures and cues to get the class’s attention. 

2 . Preschool PATHS would demonstrate an Impact on teachers’ social- 
emotional Instruction skills. These skills include modeling emotion identifi- 

^See Appendix E for more details about the content foci of each enhancement and how they do and do not 
overlap. 
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cation and labeling;^ creating an environment that is supportive of children’s 
emotional expression; encouraging techniques for calming down; facilitating 
social awareness such as empathy; helping problem-solve in social situations; 
and supporting children’s efforts to regain emotional control. 

3. Tools of the Mind would demonstrate an impact on teachers’ scaffolding 
of peer interactions and play. Scaffolding is a practice teachers use to sup- 
port children’s activities or responses at their current level of understanding 
while extending the activities or responses in order to help the children ad- 
vance to their next level of ability. Tools directly promotes teachers’ scaffold- 
ing skills, which include scaffolding children’s pretend play through extended 
planning and expanding or extending children’s games to be more elaborate, 
as well as scaffolding children’s interactions with each other while they are 
playing together. 

To measure the targeted teacher practices, independent observers collected information 
about teacher classroom management, teacher social-emotional instmction, and teacher scaf- 
folding in all Head Start CARES classrooms. Observations were conducted in the spring before 
implementation began and in the spring of the implementation year. Box 5.1 describes the in- 
stmment that was used to observe teacher practice. 

In the spring before implementation began, teachers in each enhancement group and the 
control group were similar for the most part, as shown in Table 5.1. Enhancement and control 
classrooms also scored similarly on baseline measures of teachers’ classroom management, 
teachers’ social-emotional instmction, and teachers’ scaffolding. Baseline data demonstrate that 
while only a little scaffolding and social-emotional instmction occurred in preschool classrooms 
generally, teachers spent a fair amount of time on classroom management. Because of this high 
level of classroom management at baseline, it was possible that it would be difficult for The 
Incredible Years to produce significant improvement in teachers’ classroom management. 

One of the challenges of assessing change in teacher practice is that teachers may change 
their practice over the course of the year even in classrooms where there is no special social- 
emotional intervention. For example, teachers may have a particularly challenging child in the 
classroom that year and change the way they manage their class to address the child’s behavior. 
Fortunately, because Head Start CARES was implemented in a well-conducted random assign- 
ment evaluation, teacher practice in classrooms where the enhancements were delivered could be 


^“Emotion identification” is the ability to comprehend an emotion that someone is exhibiting. “Emotion 
labeling” is the act of verbally naming an emotion while someone is expressing it. 
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Box 5.1 

Teacher Practice Measure 

Observations that were collected through an instrument called the Adapted Teaching Style 
Rating Scale (Adapted TSRS) provide information about teacher practice. In the spring be- 
fore the start of Head Start CARES implementation and in the spring of the implementation 
year, observers who were blind to the intervention status of the classrooms observed the 
teachers and classrooms for one day. Appendix F lists all ite ms on the instalment. Teachers 
were rated on: 

1 . Classroom management: consistency/routine, preparedness, classroom awareness, 
positive behavior management, negative behavior management, and attention 
to/engagement with behavior in the classroom. 

2. Social-emotional instmction: emotion modeling, emotion expression, emotion reg- 
ulation, social awareness, social problem-solving, and provision of interpersonal 
support. 

3. Scaffolding: supporting/extending dramatic play and peer interactions. 


compared with teacher practice in the control group classrooms. In a sufficiently large study 
sample such as this one, because assignment to one of the enhancements or to the control condi- 
tion occurred randomly, any changes in teacher practice or children’s behavior that were ob- 
served were not indicative of baseline differences among teachers, but instead reflected the im- 
plementation of the social-emotional enhancements. As such, analyses compare teacher practice 
scores in the spring of the implementation year for teachers in Head Start centers where en- 
hancements were assigned with teachers in control group centers. Any differences between the 
two groups of teachers are presented below as the effect, or impact, of the enhancement. Those 
differences that are unlikely to have occurred by chance are described as statistically significant. 


"^To estimate impacts for this report, mean outcomes for each enhancement group were compared with 
corresponding means for the control group, in models pooled across all three enhancements. Models also con- 
trolled for key background characteristics, including a pretest on the outcome measure (for example, the previ- 
ous spring’s observation score). Multilevel modeling was used to account for the nested nature of the data, 
where classrooms are nested within centers and centers are nested within “blocks.” Not all the centers that were 
associated with some of the larger grantees were similar in racial/ethnic composition, part-day/flill-day status, 
and classrooms with only four-year-olds versus classrooms with students who were a mix of ages. For those 
grantees, participating Head Start centers were grouped into smaller four- or eight-center random assignment 
blocks, so that all the centers in each block were comparable across these characteristics. Fixed effects account- 
ed for the nesting of centers within blocks. 
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Head Start CARES Demonstration 


Table 5.1 

Selected Baseline Characteristics of Teachers, by Enhancement 

Incredible Years (lY) Preschool PATHS Tools of the Mind 

Difference Difference Difference 


Characteristic 

Control 

Group 

Program 

Group 

(lY vs. 
Control) 

Program 

Group 

(PATHS vs. 
Control) 

Program 

Group 

(Tools vs. 
Control) 

DemosraDhics 

Age (years) 

40.38 

41.09 

0.71 

43.48 

3.10 

43.83 

3.45 * 

Race and ethnicity (%) 

White, non-Hispanic 

28.02 

30.39 

2.37 

31.59 

3.57 

27.25 

-0.77 

African-American, non-Hispanic 

30.68 

25.77 

-4.91 

32.91 

2.24 

30.40 

-0.27 

Hispanic 

36.24 

34.05 

-2.19 

29.11 

-7.13 

34.38 

-1.86 

Other/multi raciaP 

4.54 

9.80 

5.26 

6.35 

1.81 

7.95 

3.41 

Bachelor’s degree or higher (%) 

64.69 

68.77 

4.08 

66.41 

1.72 

59.38 

-5.31 

Teacher burnout’’ 

Maslach Burnout Inventory: 

Emotional Exhaustion Subscale (0-54) 

13.28 

16.72 

3.45 * 

14.50 

1.23 

13.98 

0.70 

Teacher deuression’’ 

K-6 Depression Score (0-24) 

Teacher emotion and socialization nractices' 

2.14 

3 

3.17 

1.04 * 

3.74 

1.60 *** 

3.04 

0.90 

Social-emotional practices (%) 

Focus on academics 

2.57 

8.06 

5.49 

1.21 

-1.37 

9.22 

6.64 * 

Neutral focus 

79.63 

73.72 

-5.91 

75.34 

-4.30 

72.43 

-7.21 

Focus on social-emotional development 

17.79 

18.28 

0.50 

23.45 

5.67 

18.37 

0.58 

Mean of emotion coaching items” (0-4) 

3.61 

3.41 

-0.20 ** 

3.50 

-0.11 

3.55 

-0.06 

Sample size’’ 

77 

77 


77 
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SOURCE: MDRC calculations based on the spring lead teacher self-survey (conducted at baseline). 

NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

““Other” includes Asian, Native Hawaiian/Pacific Islander, and American Indian/Alaska Native. 

’’Scale (range of scores) is shown in parentheses. 

‘’Emotion coaching was defined as teachers’ ability to positively support children’s navigation of negative or difficult emotions. 
‘’For all variables in the table, data are available for at least 90 percent of the sample. 



• As expected, teacher practice improved in classrooms where the en- 
hancements were implemented compared with the control group 
(“business as usual”) classrooms. 

Each enhancement changed the skill it emphasized, as hypothesized, and it is extremely 
unlikely that this pattern of effects would be seen by chance. Despite hypotheses that these 
teacher practices were theorized to change, it was not at all clear before implementation that 
changes in teacher practice would actually occur as intended. In fact, all three enhancements 
affected the teacher practice they emphasized, with moderate to large effect sizes, as shown in 
Table 5.2. In addition. The Incredible Years also had a small effect on social-emotional instruc- 
tion. See Box 5.2 for an explanation of effect sizes. 

The Incredible Years enhancement increased classroom management by 0.44 standard 
deviation and social-emotional instmction by 0.30 standard deviation — reflecting statistically 
significant, moderately sized effects compared with other studies of early childhood interven- 
tions,^ as explained in Box 5.2. In other words, teachers implementing the Incredible Years en- 
hancement were better able to positively manage behavior and provide social-emotional instmc- 
tion to children compared with teachers in the control group, who had an average score of 3.79 
(out of a possible 5) on classroom management. In comparison, teachers in the Incredible Years 
group had an average score of a 4.09 on classroom management. Although classroom manage- 
ment was already relatively strong in the control classrooms, teachers in the Incredible Years 
group were still able to strengthen their classroom management practices further. One lead 
teacher described in detail how she used various Incredible Years behavior management strate- 
gies to address two children’s different behavioral needs: 

1 had two students on behavior plans. One, we used the ignoring with the un- 
wanted behavior, which was all the whining. That whining has vanished. That 
was wonderful. For the other guy, this fellow was touchy. He was nonchalant. 

He didn’t care about anything. He wasn’t going to do his work or anything. With 
him, we did the extra praising. We would catch him doing something well, we 
would praise and encourage. Now, he’s writing, he doesn’t meddle. 1 owe that to 
Incredible Years. That was an incredible thing. Before 1 was thinking, wow 1 
want to help him, but 1 don’t know how. 

The PATHS enhancement increased social-emotional instruction by 0.92 standard de- 
viation, which reflects a statistically significant, relatively strong impact. PATHS teachers 
were better able than their control group counterparts to model the appropriate identification of 
emotions and use of social problem-solving techniques in challenging social situations. A 


^Raver et al. (2008, 2011); Bierman et al. (2008a); Morris et al. (2010); Domitrovich et al. (2009). 
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Head Start CARES Demonstration 


Table 5.2 

Classroom-Level Ratings and Impacts, by Observation of Teacher Practices 


Teachers’ Practice 

Control 

Group 

Rating 

Incredible Years (lY) 


Preschool PATHS 


Tools of the Mind 


Program 

Group 

Rating 

Difference 

(lY vs. Effect 

Control) Size“ 

Program 

Group 

Rating 

Difference 
(PATHS vs. 
Control) 

Effect 

Size* 

Program 

Group 

Rating 

Difference 
(Tools vs. 
Control) 

Effect 

Size“ 

Classroom management 

3.79 

4.09 

0.30 

0.44 

3.90 

0.12 

0.17 

3.89 

0.10 

0.15 

Social-emotional instruction 

1.76 

1.98 

0.22 * 

0.30 

2.42 

0.66 *** 

0.92 

1.78 

0.02 

0.02 

Scaffolding 

1.44 

1.41 

-0.03 

-0.06 

1.48 

0.05 

0.09 

1.78 

0.35 

0.68 

Sample size'’ 











Head Start center 

26 

26 



26 



26 



Classroom 

77 

77 



77 
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SOURCE; MDRC calculations based on observational assessments completed using the Adapted Teaching Style Rating Scale (Adapted TSRS). 


NOTES; Statistical significance levels are indicated as follows; *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

All models are based on pooled analyses of program group status, controlling for the pretest and blocking variable used to randomly assign 4, 8, or 12 
centers to the program group. 

Teachers’ practice was rated on a scale of 1 (low) to 5 (high), reflecting fidelity of classroom implementation. 

^Effect size is calculated by dividing the impact of the program (the difference between the means for the program group and the control group) by the 
standard deviation for the control group. 

’’For all variables in the table, data are available for 100 percent of the sample. 



Box 5.2 

Understanding and Contextualizing Effect Sizes 

Effect sizes provide a way to compare the findings in Head Start CARES with findings from 
other studies to determine the magnitude of the impacts across different measures. The effect 
size defines the proportion of a change that takes place following implementation of the inter- 
vention — specified by change in the standard deviation, or variation, of the outcome — that 
can be attributed to the intervention. 

Bloom and colleagues suggest that the magnitude of effects in educational interventions can be 
understood by comparing the size of the effects in similar policy-relevant contexts.' Other pre- 
school interventions that have focused on early childhood social-emotional development have 
been smaller in scale, have had richer resources, and have been more intensive than the en- 
hancements that were implemented in Head Start CARES. For instance, in the Foundations of 
Learning demonstration and CSRP (formerly known as the Chicago School Readiness Pro- 
ject),^ highly credentialed and clinically licensed coaches provided a full day of consultation 
each week for teachers, at times directly intervening one-on-one with children. In the Head 
Start REDl Program,^ another early childhood intervention, teachers received support for both 
the social-emotional and literacy domains. In these more intensive early childhood interven- 
tions, effect sizes on teacher practices of 0.50 were considered moderate, and effect sizes above 
0.80 were considered large.^ In Head Start CARES, although more widely scaled up and less 
intensive, the first-order effect sizes are still moderate to large, ranging from 0.30 to 0.92. 


'Bloom, Hill, Black, and Lipsey (2008). 

'^Raver et al. (2011). CSRP is not associated with The Chicago School®, which is a trademark of 
The Chicago School of Professional Psychology 

^Bierman et al. (2008a). 

%or instance, in the Foundations of Learning demonstration, for pre-kindergarten classroom man- 
agement impacts, the teacher practice effect sizes that were measured using the Classroom Assessment 
Scoring System (CLASS) were moderate (0.46 for teacher sensitivity) to large (-0.90 for negative cli- 
mate); see Morris et al. (2010). In CSRP, CLASS impacts ranged from 0.52 to 0.89 (Raver et al., 2008). 
In REDI, CLASS impacts ranged from 0.39 to 0.61 (Domitrovich et at, 2009). These impacts on teach- 
er practice were also sufficiently large to lead to impacts in child outcomes (Morris et al., 2010). 


PATHS coach described how she saw teachers change their behavior to model appropriate so- 
cial interactions: 

The teachers are doing emotion coaching, using the term “friends.” At this point 
it feels very natural for the teachers. The children seem to respond well to what 
the teachers are doing, but the teachers are the driving force. 
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In fact, the coach ascribed the changes she saw in the classroom directly to the teachers’ 
ability to change their practice. And it seems that there is room to change such practice: teachers 
in the PATHS group had an average social-emotional instmction score of 2.42 (out of a possible 
5) in comparison with the control group’s average score of 1.76. Given the low levels of social- 
emotional instmction in control group classrooms and the capacity for teachers to change nearly 
a full standard deviation over the course of a year, this practice seems to be a particularly malle- 
able teacher skill. 

Finally, the Tools of the Mind enhancement increased teacher scaffolding by 0.68 
standard deviation, also statistically significant and relatively strong. Looking at the control 
group mean of 1 .44, it is clear that teachers generally do some but not much scaffolding of chil- 
dren’s play and peer interactions, in comparison with the program group’s average score of 
1.78. The Tools enhancement helped teachers provide more scaffolding for children as they 
played, demonstrating that scaffolding is a changeable practice that teachers can learn. A lead 
teacher using Tools of the Mind described how difficult it was to scaffold children to self- 
regulate and to resist playing with a toy: 

[l]n pretend play, we used dinosaur figurines to help to tell the story, and it was 
hard for the children to refrain from playing with them right away. We have to 
teach the children about their boundaries and limitations, and that was just chal- 
lenging. 

The moderate to large “first order’’ impacts on directly targeted teacher practices and 
skills are an encouraging first step in confirming the Head Start CARES theory of change. They 
also demonstrate that, even with a more geographically dispersed implementation support sys- 
tem than was provided for the efficacy trials of the enhancements,*’ it is possible to change 
teacher practice at scale. However, it is important to note that a support system was put in place 
to achieve implementation at this level. As hypothesized, teachers implemented the enhance- 
ments with quality, and teacher practices also changed as a result. A future report will examine 
the next steps in the theory of change — teacher-child interactions and children’s outcomes. 


Summary 

Changing an individual’s behavior and day-to-day practices is not an easy task. These promis- 
ing results demonstrate that, with the necessary support and attention, teachers can implement 
sometimes complex interventions that lead to change in their classroom practices. The imple- 
mentation of the Head Start CARES enhancements and the resulting changes in teacher practice 
represent an important first step in better supporting the development of children’s social- 
emotional competencies. 

*’Domitrovich, Cortes, and Greenberg (2007); Webster-Stratton, Reid, and Hammond (2001); Barnett et al. 
(2008). 
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Chapter 6 

Head Start CARES: Findings and Lessons About 
Implementation and Scale-Up 


This chapter summarizes the Head Start CARES findings thus far, as well as practical lessons 
on the implementation and scale-up of social-emotional enhancements of the type tested in the 
Head Start CARES demonstration. 


Main Findings 

Overall, Head Start CARES provided a fair test of three social-emotional interventions, or 
Head Start program “enhancements,” which were intended to enrich and complement existing 
Head Start practices and curricula: The Incredible Years Teacher Training Program, Preschool 
PATHS (Promoting Alternative Thinking Strategies), and Tools of the Mind — Play. Teachers 
were randomly assigned among the enhancements, and all teachers in the study sample had 
similar characteristics when the study began. Coaching, training, and ongoing monitoring were 
also designed to be generally the same across the enhancements. The support system that was 
put in place to bolster expansion, or “scale-up,” of the intervention — from the initial small 
efficacy trials to delivery in 104 Head Start centers — was implemented as intended across all 
three enhancements. 

The enhancements were implemented at or above the threshold level of satisfactory fi- 
delity — meaning that they were implemented as intended — albeit with some variation across 
all three of them. While The Incredible Years and Preschool PATHS were implemented at 
moderately high levels of fidelity to the original design. Tools of the Mind was implemented on 
a weaker, but still satisfactory, level. Because teachers and the support system were similar 
across the three enhancements, it is likely that the characteristics of the enhancements them- 
selves — and not differences among teachers, coaching, or training — were the reason for vary- 
ing levels of implementation. Despite this variation, all three enhancements reached or exceeded 
the threshold for satisfactory fidelity, leading to moderate to strong effects on the hypothesized 
changes in teacher practices. Despite the magnitude of the effects, the absolute levels of teacher 
practice for both social-emotional instmction (the emphasis in Preschool PATHS) and scaffold- 
ing (the emphasis in Tools of the Mind) were still low, below a rating of 3 on a scale of 1 (low) 
to 5 (high). 

These findings reflect the first of three steps in a theory of change for the Head Start 
CARES demonstration. The theory of change specifies how the enhancements are hypothesized 
to change child outcomes at the end of the preschool year. The first step hypothesizes that im- 
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plementation of the enhancements with fidelity to the original design will strengthen some ex- 
isting teacher practices and lead to changes in other practices. As seen in this report, the en- 
hancements were indeed implemented with fidelity, and this led to the expected changes in 
teacher practice. 

The next two steps in the theory of change hypothesize that the changes in teacher prac- 
tice will lead to improved classroom interactions and that those changes in classroom interac- 
tions will, in turn, lead to improved child outcomes. This report confirms that the enhancements 
were implemented with the level of fidelity needed to provide a fair test of whether the next two 
steps in the theory of change will result. A separate report on Head Start CARES examines the 
impacts of the three enhancements on teacher-child interactions and children’s outcomes.' 


Lessons Learned 

The remainder of this chapter draws on information that was collected during site visits, as well 
as the multiple data sources used in the Head Start CARES demonstration, and builds on the 
findings from earlier chapters in this report to present important considerations for the imple- 
mentation of social-emotional enhancements. Although the demonstration was conducted in 
Head Start classrooms, the lessons are also relevant to other early childhood programs. Never- 
theless, the application of these lessons should be considered in light of the specific context of 
each particular program. 

The effort to implement the Head Start CARES enhancements with fidelity appeared 
to benefit in particular from (1) a comprehensive professional development model that in- 
cluded training and coaching, (2) ongoing monitoring and technical assistance, (3) well- 
articulated enhancements, and (4) organizational “buy-in” and support. Each of these factors 
is discussed below. 

Comprehensive Professional Development 

Training and coaching were valuable and complementary parts of the Head Start 
CARES professional development model. Training built a base of understanding from which 
coaches and teachers could work, while coaches expanded upon and individualized the infor- 
mation they received in the training. Teachers and coaches reported that the professional devel- 
opment facilitated implementation of the enhancements. 


'Morris et al. (forthcoming). 
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In order to scale up an enhancement, particular attention should be paid to maintaining 
the “dosage” — or frequency and duration — of professional development, which can be easily 
diluted because of challenges in hiring and retaining trainers and coaches, or because of obsta- 
cles to teacher attendance at training sessions. Enough trainers should be available to provide 
multiple training days throughout the year, and teachers should be urged to attend all training 
sessions. There should also be enough coaches to provide teachers with regular meetings and 
classroom observations, and time and resources should be set aside to allow teachers to meet 
with coaches. High-quality professional development requires certified and skilled trainers who 
understand the enhancements and can train teachers to use them, as well as coaches who are 
expert in the enhancement and have the skills to help teams of teachers learn. Overall, the pro- 
fessional development model benefited from a professional workforce of high-quality trainers 
and coaches, and was an important component of the intervention. 

Training 

• Teachers implementing social-emotional interventions appear to benefit 
from ongoing training. 

One-time workshops for teachers have been found to be ineffective in supporting im- 
plementation of new practices in early childhood settings; research suggests that more intensive 
training may lead to improved teacher outcomes.^ In Head Start CARES, training was provided 
throughout the year. Teachers benefited from learning strategies in the training sessions, practic- 
ing them in the classroom, and then returning to training to discuss the implementation process. 
Training that occurred throughout the year not only enabled teachers to learn new content and 
build upon content that they had learned previously, but also promoted continuous feedback and 
support about the enhancement practices from trainers, coaches, and the teachers’ peers. 

• Training lead and assistant teachers together can improve implementa- 
tion and sustainability, but it is resource-intensive and requires careful 
planning. 

Eead and assistant teachers reported that being trained together strengthened their rela- 
tionship and helped bolster implementation. In addition, training together seemed to empower 
assistant teachers to play an active role in implementation. (It was particularly helpful to have 
joint training for Tools of the Mind, which was time-intensive and required both teachers to ac- 
tively participate in the implementation process.) However, joint training requires careful plan- 
ning to address issues such as space, classroom coverage, and additional materials and transpor- 
tation needs. 


^Wasik, Mattera, Lloyd, and Boiler (2013). 
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Attendance was aided by holding training on designated “professional development 
days” when possible and paying teachers to attend training when a professional development 
day was not available. In addition, training sessions that were conducted off site or required 
travel were more difficult for teachers to attend than those that were nearby. Therefore, training 
planners should consider whether the sessions can be held on site, take place during existing 
professional development days, and include compensation for teachers who attend. 

Coaching and Coaches 

• Coaches need to be hired and trained before the school year starts. 

In a few cases in Head Start CARES, coaches did not start the school year and training 
sequence with the teachers. This made it difficult to complete the coaches’ training and to build 
their rapport with teachers. In fact, coaches should be trained ahead of teachers in order to de- 
velop their expertise in the content on which they will be coaching. While pre-training for 
coaches can be logistically challenging and costly, both teachers and coaches believe that it is 
important. 

In order to train coaches on time, they should be hired before the start of the school 
year. Depending on geography, the number of coaches needed, and the type of skills required, 
the recmitment and hiring process can be lengthy, so it should be started early. In Head Start 
CARES, this meant at least three to four months in advance of the start of implementation (and 
perhaps even earlier if coaches were to be fully trained ahead of time). 

• The coaches should reflect the needs of the particular organization and 
population that the organization serves. 

Each organization serves a unique population with its own needs. During the hiring 
process, careful thought should be given to whether the coaches’ abilities and credentials match 
the needs of the students and teachers. For example, in Head Start CARES, the coaches had to 
speak the language used in the classrooms in order to support implementation. 

• Coaches need supervision and support. 

The coaches in Head Start CARES needed supervision and support in multiple areas, 
including both the more typical administrative and logistical support as well as supervision to 
ensure high-quality implementation of the enhancement. Coaches received expert support from 
the trainers on enhancement content. From the grantees, coaches needed help with building and 
maintaining teacher rapport and enthusiasm, along with support for logistical matters such as 
salary payments and performance reviews. Depending on the intervention and process for im- 
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plementation, supervision can come from only one source or from multiple sources, as in Head 
Start CARES. Either way, coaches will need support for both content and logistics. 

Ongoing Monitoring and Technicai Assistance 

The collection and monitoring of timely implementation data were cmcial to the quick 
and comprehensive provision of technical assistance. Monitoring implementation in the class- 
rooms allowed developers to work better with trainers and coaches on how to support teacher 
practice, and allowed the technical assistance team to address any logistical needs, hnplementa- 
tion of social-emotional enhancements, particularly at scale, would benefit from a monitoring 
system that is similarly able to communicate the data to key stakeholders, including developers, 
trainers, coaches, and centers. 

• A designated entity is needed to oversee implementation, including an 
ongoing monitoring effort that uses a flexible management information 
system (MIS). 

The Head Start CARES MIS was an online database that proved invaluable for under- 
standing and supporting implementation through ongoing data monitoring and the provision of 
technical assistance. The MIS was important for collecting data, reviewing implementation, dis- 
seminating information about implementation to grantees, and supporting coaches and trainers 
in their work. In Head Start CARES, because of the widespread implementation across multiple 
locations, MDRC operated the system. However, whether in smaller-scale or more widespread 
implementation, a designated person or group should collect data, monitor implementation, and 
plan technical assistance based on the data. In Head Start CARES, dosage and quality were 
monitored against predetermined standards or thresholds, which were used to flag classrooms 
that might need support. This monitoring and technical assistance process does not need to be 
external; it could be provided at the local level by a member of the participating organization. 
Regardless of how the monitoring is organized, it should trigger immediate assistance to sfrug- 
gling classrooms. 

Well-Articulated Enhancements 

The three enhancements in the Head Start CARES demonstration each had a different 
focus and flavor. Implementation of these or other social-emotional enhancements needs to take 
into account a number of issues that arise from the characteristics of the enhancement itself, as 
well as how the enhancement fits into the operations and philosophy of the host organization 
and its teachers. 

• It is important to identify and communicate which components of the 
enhancement are most critical. Enhancements that specify more con- 


85 



Crete components, actions, and lessons may be easier for teachers to im- 
plement than less scripted enhancements. 

Although the Head Start CARES enhancements were all well documented, teachers still 
stmggled with the sheer amount of prepared manuals, materials, and expectations. Articulating 
which of the many activities, lessons, and processes were the most critical to implement helped 
teachers and coaches know where to focus their ahention. For example, over the course of the 
implementation of Preschool PATHS in the Head Start CARES demonstration, PATHS devel- 
opers articulated even more clearly — that is, beyond the contents of the prepared materials and 
documentation — their vision for fidelity to the original design, focusing on one PATHS lesson 
and one extension activity a week. 

In addition, the type and quality of the materials mattered. Similar to findings in other 
studies,^ activities that were well scripted and had clear and prepared materials were easier to 
implement than activities that gave teachers a set of directions but no scripted activities. The 
availability of clear, concrete language and materials to use when practicing a new skill or les- 
son with children seemed to boost teachers’ confidence and make them more receptive to im- 
plementing the enhancement. 

• Selection of enhancements should take into account how well they fit 
with an organization’s core curriculum and program philosophy. 

Some grantees and centers had clearer and more established philosophies and objectives 
than others did. Organizations with a well-defined philosophy that fit well with the enhance- 
ments’ core components were able to implement the intervention more effectively. Enhance- 
ments that aligned with grantees’ philosophies appeared to improve buy-in and support for im- 
plementation. 

However, grantees that implemented the enhancements well, regardless of which en- 
hancement, were very clear about the philosophy of their organization, which helped teachers 
and coaches to better understand what could be implemented. If any components or activities 
ran counter to an organization’s mission, teachers and coaches could then actively address how 
to adapt those pieces while still maintaining the intent of the enhancement. For example, many 
grantees already had curricula or assessments in place that teachers had to complete. Additional- 
ly, sometimes the participants believed that the enhancements conflicted with Head Start re- 
quirements. Whether the grantee completely removed particular obligations from teachers or 
stated clearly which obligations should be priorities, the teachers were able to deliver the inter- 


^Durlak and Dupre (2008); Fixsen et al. (2005). 
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vention with greater ease and assurance once the grantee explained how to balance its curricula 
with the Head Start CARES enhancements. 

Organizational Capacity and Support 

Implementation of the enhancements with fidelity required sustained attention and sup- 
port from Head Start leadership. 

• Organizational capacity is important for delivering high-quality training 
and coaching and supporting implementation of the enhancements. 

In order to support implementation, grantees need diverse capacities, including the abil- 
ity to hire coaches with appropriate skills, hire coaches on time (that is, before the school year 
begins), send clear messages of support for implementation to teachers and administrators, pro- 
vide space and time for teachers to participate in professional development, supply the needed 
resources to classrooms to implement the enhancements, and be continually involved in and 
supportive of implementation. 

• Grantee administration should provide clear and consistent messages 
about the importance of the enhancement. Implementation of the en- 
hancement needs to be a high priority. 

The Head Start CARES demonstration revealed that when organizations supported the 
enhancements, they sent clear messages indicating that implementation should be a priority. In 
addition to clear and consistent messaging, grantee administrators should work with coaches 
and teachers to remove barriers to implementation, provide needed resources for the class- 
rooms, and facilitate the training, coaching, and classroom implementation process. Additional- 
ly, particular challenges arose when the enhancements were implemented while Head Start pro- 
grams were also under review, implementing other assessments, or focusing on other initiatives. 
Although early childhood programs will always have other responsibilities and requirements, 
making implementation of the enhancement a high priority is cmcial to gaining the necessary 
backing from teachers and administrators. 

• Potential conflicts between the enhancement and additional require- 
ments need to be acknowledged and addressed early in the implementa- 
tion process. 

It is possible that certain aspects of the enhancement will conflict with existing pro- 
gram requirements, such as assessments, curricula, or rules. For example, in the Head Start 
CARES demonstration, some grantees were concerned that using food as a reward did not 
align with Head Start guidelines. For those grantees, acceptable alternative ways to provide 
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positive reinforcement needed to be found. Developers and trainers worked closely with grant- 
ees on such conflicts to ensure that the enhancements were appropriately adapted and still im- 
plemented with fidelity. 


Summary 

The three Head Start CARES program enhancements, in conjunction with ongoing training and 
coaching for teachers, were implemented at scale with fidelity to the original design at or above 
the predetermined threshold rating of 3 (satisfactory) out of a possible 5. While fidelity varied 
by enhancement, with The Incredible Years and Preschool PATHS implemented at a moderate- 
ly high level of fidelity and Tools of the Mind implemented on a weaker but near-satisfactory 
level, each enhancement led to the expected changes in its targeted teacher practices. Despite 
the challenges with implementation, this study demonstrates that, with the appropriate supports, 
it is possible to implement social-emotional enhancements at scale in Head Start. 



Appendix A 

Head Start CARES Sample and Selection 
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The selection of grantees/delegate agencies (hereafter referred to as “grantees”) was a multistep 
process in which data were first used from the Program Information Report (PIR) database to 
create a sample of grantees, and then information from the grantees themselves was used to 
further constrain the sample. Much of the data needed to make selection decisions required 
direct contact with the grantees. 

The sample was selected from the 2006-2007 PIR database of all Head Start grantees, 
including Early Head Start grantees. ' From this database, a population of Head Start grantees 
was defined.^ Grantees were excluded from the sample if they met any of the following criteria: 
(1) they were Migrant and Seasonal Head Start grantees;^ (2) organizations providing non- 
center-based care (for example, home care); (3) they were located within the U.S. territories, 
Alaska, and Hawaii; and (4) they were more than 1 00 miles from a “primary airport.”'* 

A number of these exclusions have been imposed by previous national research studies 
of Head Start, such as the Head Start Family and Children Experience Survey (FACES)^ and the 
Head Start Impact Study.*’ Two additional exclusions were used in the Head Start Impact Study, 
but were not imposed by the Head Start CARES study: (I) grantees operated by tribal organiza- 
tions, and (2) grantees in saturated communities.’ The FACES sampling frame has also excluded 
grantees mn by tribal organizations, as well as grantees in Puerto Rico and other territories. 


'Early Head Start is a federal program that provides early, continuous, intensive, and comprehensive child 
development and family support services to low-income infants and toddlers and their families, and to pregnant 
women and their families. 

’American Indian grantees were included. 

’Migrant and Seasonal Head Start (MSHS) programs, which specifically respond to the needs of migrant 
farm worker families, were excluded because of the unique characteristics of the programs (many operating for 
shorter school years and at different times of the year) and the families they serve. MSHS programs were 
included in a parallel study. The MSHS CARES study is a theory-based study to explore how a social- 
emotional intervention included in the Head Start CARES study may best serve the needs of MSHS programs 
and classrooms. Implementation findings of the study will be published in 2014. 

'*The research team wanted to ensure that travel time and distance were reasonable from airports that sup- 
ply sufficient service, both for training and/or site visits to the grantees, and for teachers from the selected 
grantees to travel to chosen training hubs. A primary airport is one that has more than 10,000 passenger 
boardings per year. 

’Moiduddin et al. (2012). 

*’In the past, the FACES sampling frame also excluded Early Head Start and migrant grantees. The Head 
Start Impact Study excluded migrant grantees, grantees in U.S. territories except for Puerto Rico, grantees that 
were extremely new to the program (in operation for approximately less than two years), grantees that were 
substantially out of compliance with Head Start Performance Standards, and grantees that were underenrolled. 

’The Head Start Impact Study stratified its sample based on how saturated the early childhood education 
market was in a given area. In other words, on the basis of existing “non-Head Start comprehensive services” 
(as a function of the availability of state-funded pre-kindergarten programs and related services) — a very 
important variable for that study because children were randomly assigned, and the availability of non-Head 
Start comprehensive services would affect the experiences of control group children who were not selected for 
treatment. Because the Head Start CARES demonstration randomly assigned Head Start centers within 

(continued) 
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Next, all grantees in the sampling frame were stratified into 24 strata according to a 
combination of the following factors: (1) the region of the country in which they were located 
(Northeast, South, Midwest/Plains, and West); (2) the racial/ethnic composition of their child 
enrollment (predominantly African-American, predominantly Hispanic, or other);* * and (3) the 
“urbanicity” of their location (metropolitan or nomwetropolitan).'^ In formation on the population 
of children served within each of these 24 strata was used to determine the number of grantees 
to target for recmitment within each stratum. For example, 35,665 children, or 4.6 percent of all 
enrolled children, were located in grantees in the Northeast, in the third race/ethnicity concen- 
tration category of “Other,” and in metropolitan areas. Candidate grantees were then randomly 
sampled from within each stratum. The sampling was weighted based on child enrollment such 
that larger grantees were more heavily weighted. 

Before screening, grantees were excluded if (1) they had been in operation for less than 
two years; or (2) they were not in compliance with Head Start Program Perfomiance Stand- 
ards. Finally, recmitment teams screened selected grantees from within each stratum. Addi- 
tional exclusion criteria were drawn from the contact that recmitment teams had with grantees. 
Grantees were excluded from the sample if (1) they had fewer than four centers, or (2) they 
were already systematically implementing a social-emotional curriculum or participating in 
another major research initiative. In addition, classrooms were deemed eligible if they had only 
four-year-olds or mixed ages; ' ' centers needed to have two or more of these eligible classrooms 
to participate in the study. Once the recmitment team had spoken with grantees and collected 
information about the exclusion criteria, the selection of grantees was prioritized and narrowed 
for further recmitment based on screening and randomization criteria. In addition, grantees in 
regional hubs were prioritized to reduce training and data collection costs. 


grantees to treatment or control group status (thus, all children in the treatment and control groups were in a 
Head Start program), it is not necessary for Head Start CARES to stratify based on the availability of alterna- 
tive services. 

*RaciaFethnic composition was defined by child enrollment in the grantee, with one of three possible defi- 
nitions: grantees serving predominantly Hispanic children, grantees serving predominantly African-American 
children, and grantees serving a mix of children with various raciafethnic backgrounds. 

^Urbanicity is based on the Beale Code, a widely used geographic code developed by the U.S. Department 
of Agriculture (2004). Codes are calculated by examining the size of a county and its proximity to a metropoli- 
tan area. See www.ers.usda.gov/briefing/rurality/RuralUrbCon for more detailed information about this coding 
system. 

*°New programs or programs that were not in compliance with Head Start Program Performance Stand- 
ards were excluded from sampling because they were considered at higher risk for closing. 

"The main focus of the Head Start CARES demonstration was on four-year-old children because most of 
the evidence base for the selected enhancements focused on children who were four years of age. Classrooms 
that had all three-year-olds or had a majority of three-year-olds would not provide a sufficient number of four- 
year-olds for the sample size needed. A separate report will focus on a more limited and exploratory set of 
questions about the effect of social-emotional interventions on three-year-olds. 
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In the Head Start CARES demonstration, an online database known as the management 
information system (MIS) was used to collect systematic implementation data and to help a 
centralized technical assistance team to monitor implementation in real time. The MIS allowed 
the technical assistance team to monitor implementation of coaching, coach supervision, and 
classroom implementation of the enhancements, and to intervene with additional support when 
needed. The technical assistance team used MIS data to generate monthly reports, which they 
used to assess whether classrooms and coaches were meeting pre-specified benchmarks. MIS 
data were also used as implementation research data. 

The MIS collected data from coaches and trainers using seven different insfruments. 
Coach data sources included (1) a one-time summer checklist documenting steps taken before 
the start of the school year to work with teachers, (2) coach weekly logs documenting their 
classroom observations and contact with teachers, (3) coach monthly fidelity logs documenting 
fidelity to the enhancement as delivered in the classroom, and (4) one-time coach end-of-year 
reflections documenting each coach’s acceptance of the enhancement and challenges encoun- 
tered throughout the year.' Trainer data sources included (1) trainer supervision logs document- 
ing contact with coaches, (2) trainer fidelity logs assessing fidelity in the classrooms visited, and 
(3) trainer logs of coach quality that documented the trainer’s assessment of coach effective- 
ness." 


Monitoring Log Compietion and Data 

“Rolling logs’’ were logs that were submitted on a regular basis (such as the coach weekly log, 
the coach monthly fidelity log, or the trainer supervision log). The forms were processed as they 
were completed, and reports based on these data were shared with the technical assistance team 
on a regular basis. For example, as previously mentioned, a technical assistance threshold was 
created for the coach monthly fidelity log to indicate satisfactory implementation. On a scale of 
1 (low) to 5 (high), classrooms with a monthly coach rating above a 3, meaning that teachers 
were implementing the enhancement occasionally but not consistently, were considered to have 
reached the threshold for satisfactory implementation. Classrooms that did not reach or dropped 
below the threshold received additional monitoring and technical assistance. These monitoring 
efforts focused on distributing information to the various key parties involved in Head Start 
CARES as it was collected. 


'Coach end-of-year reflections and the summer checklist were completed only for Cohort Two, at the end 
of their implementation year. 

"For a Ml description of all the data sources used in Head Start CARES implementation, see Appendix C. 
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Examples of MIS Data Sources 

In order to provide a sense of the MIS, this appendix includes items and instmctions from the 
coach weekly log and two sections of the coach monthly fidelity log data sources. Much of the 
time, only these items were provided in the MIS, and the fuller set of documentation and 
instmctions were provided to coaches and trainers at the kick-off meeting that was held at the 
beginning of the school year. 
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Coach Weekly Log 


Detailed Explanations of Coach Weekly Log Questions 

1. How much time did you spend meeting with your teachers since your last coaching 
session, including the time spent in this week’s coaching session? (Select minutes and 
hours.) 

This question asks you to record how much time you met with your teachers since the last 
coaching session. Usually, this will only include the time you spend in your weekly coach- 
ing session, debriefing, solving problems, goal-setting, or discussing other issues relevant to 
coaching or teaching as part of Head Start CARES. However, sometimes a coach will meet 
with a teaching pair or individual teacher for an additional amount of time. If you do, please 
also include this time in your total. In general, we recommend that you meet with each 
teaching pair for at least 30 minutes each week, although you may spend more time in 
classrooms that need additional assistance. 

2. How much time did you spend in the classroom doing classroom coaching, modeling, 
or observing since your last coaching session? (Select minutes and hours.) 

This question asks you to record how much time you spent in the classroom, doing class- 
room observing or modeling, since the last coaching session. In general, you should be 
spending 1.5 hours in each classroom each week. This time should include some active 
modeling and demonstration for teacher and children in the activities of the program en- 
hancement, as well as observation. 

3. How much time since your last coaching session did you spend preparing for your 
observation or this week’s coaching sessions? (Select minutes and hours.) 

This question asks you to record how much time you spent preparing for your observation 
or weekly coaching session, including time spent getting materials ready, reviewing teach- 
ers’ progress, and completing and submitting weekly and monthly logs. In general, you 
should be spending one hour per week per classroom preparing for your coaching sessions. 
Weeks that include the preparation and submission of coach monthly fidelity logs may re- 
quire more time, while other weeks may require less of a time commitment. 

4. According to the teachers, since your last coaching session, to what extent were they 
able to use program strategies, lessons, activities or concepts? 

During your coaching session, we ask that you discuss with your teachers the extent to 
which they were able to use program strategies, lessons, activities, or concepts. 

• If they say they haven’t used them at all, select “1 = Not at all." 

• If they say they have used them a very small amount (for example, one activity for a 
few days, or a few activities on only one day), select “2. ” 
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• If they say they are using the majority of the activities they have been asked to do, but 
not all of them, or not consistently, select “3. ’’ 

Finally, if they say they are using nearly all of the program activities nearly or every 
day, select “4 = A lot. " 

5. What coaching strategies did you use with these teachers in this coaching session? 
(Check all that apply,) 

For this question, we would like to know what you have covered in your coaching session 
with your teachers. Please check all that occurred in your coaching session. 

1“ 1 - Answered question about the program 

V 2 - Demonstrated strategies or techniques 

I” 3 - Role-played strategies 

I” 4 - Showed and discussed video examples 

I” 5 - Provided feedback using notes or data from observation 

1“ 6 - Problem-solved: individual child 

V 7 - Problem-solved: classroom situation 

V 8 - Problem-solved: center situation 

I” 9 - Reflective discussion 

I” 10 -Goal setting 

V 1 1 - Planned for what the teacher is going to do in the classroom 
I” 12 - Set a date for the next observation and coaching session 

6. Overall, how productive was this eoaching session? 

This question asks about the overall productiveness of the session, in terms of how much 
was accomplished in the session, whether you feel the teachers were receptive to what was 
discussed, and whether what you discussed will help the teachers in the coming week. If: 

• Nothing was accomplished, in terms of discussing strategies or techniques, goal-setting, 
problem-solving, and planning for the coming week, and the teachers were unrespon- 
sive to the session, select “1 = Very Unproductive. ” 

• Very little was accomplished, in terms of discussing strategies or techniques, goal- 
setting, problem-solving, and planning for the coming week, and the teachers were 
mostly unresponsive to the session, select “2. ” 
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• Only a little was accomplished, in tenns of discussing strategies or techniques, goal- 
setting, problem-solving, and planning for the coming week, and the teachers were only 
partially engaged, select “3. ” 

• You were able to discuss appropriate strategies or techniques, set goals, problem solve, 
and plan for the coming week, and the teachers were responsive to the session, select 
“4. ” 

• The session went exceptionally well, and you and the teachers worked together in an 
energetic and productive manner, discussing appropriate strategies or techniques, set- 
ting goals, problem solving, and planning for the coming week, and the teachers were 
very responsive to the session, select “5 = Very Productive. ’’ 
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Coach Monthly Fidelity Report: General Fidelity Items^ 


Modeling and Generalization of the [Specific Enhancement] 

1 . It is clear when you enter this classroom and look around it is a well-managed classroom, 

characterized by the [specific enhancement] strategies taught in [specific enhancement], 

• If the classroom has no tangible signs of the [specific enhancement] strategies, select “1 
= strongly disagree. ” 

• If the classroom has only one tangible sign of [specific enhancement] strategies, select 
“ 2 . ” 

• If the classroom has a few tangible signs of [specific enhancement] strategies, select 
“3. ” 

• If the classroom has many tangible signs of the [specific enhancement] strategies, select 
“4. ” 

• If the classroom is an “exemplary” [specific enhancement] classroom, that is, there are 
numerous tangible signs of [specific enhancement] strategies, select “5 = strongly 
agree. ” 

2. The teachers have taken extra steps to extend the [specific enhancement] strategies and 

concepts into other parts of the Head Start program by designing special activities or adapt- 
ing standard activities to be consistent with [specific enhancement] themes. 

• If the teachers are actively opposed to taking extra steps to extend the [specific en- 
hancement] strategies and concepts into other parts of their classroom activities and 
schedule, select “1 = strongly disagree. ” 

• If the teachers have not taken steps to extend the [specific enhancement] strategies and 
concepts into other parts of their classroom activities and schedule, select “2. ” 

• If the teachers have attempted to extend the [specific enhancement] strategies and con- 
cepts, but have mostly not been successful, select “3. ” 

• If the teachers have taken extra steps to extend the [specific enhancement] strategies 
and concepts into other parts of their classroom activities and schedule and have been 
successful, select “4. ” 

• If the teachers have gone “above and beyond,” successfully integrating [specific en- 
hancement] strategies and concepts into numerous parts of their classroom activities 
and schedule, select “5 = strongly agree. ” 


^These items are presented generally for all three enhancements. Some individual items may have includ- 
ed enhancement-specific examples, such as an Incredible Years, Preschool PATHS, or Tools of the Mind 
strategy. These enhancement-specific examples are not included here. 
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3. The teachers make active use of [specific enhancement] strategies in an integrated fashion 

throughout the day, not just when problems arise. 

• If you never observe teachers making use of [specific enhancement] strategies, select 
“1 = strongly disagree. ” 

• If you only observe teachers using [specific enhancement] strategies once or twice a 
day, select “2. ” 

• If you only observe teachers using [specific enhancement] strategies when problems 
arise, select “3. ” 

• If you observe teachers using [specific enhancement] strategies throughout most of the 
day, and not only when problems arise, select “4. ” 

• If teachers make frequent use of [specific enhancement] strategies nearly all day, select 
“5 = strongly agree. ” 

4. The teachers use [specific enhancement] as part of their strategies for managing conflicts, as 

part of classroom procedures, and to help build positive relationships between the children. 

• If teachers’ never use [specific enhancement] strategies for managing conflicts, as part 
of classroom procedures, or to help build positive relationships between the children, 
select “1 = strongly disagree. ” 

• If the teachers rarely use [specific enhancement] strategies for managing conflicts, as 
part of classroom procedures, or to help build positive relationships between the chil- 
dren, select “2. ” 

• If the teachers occasionally use [specific enhancement] strategies for managing con- 
flicts, as part of classroom procedures, or to help build positive relationships between 
the children, but not consistently, select “3. ” 

• If the teachers often use [specific enhancement] strategies for managing conflicts, as 
part of classroom procedures, or to help build positive relationships between the chil- 
dren, select “4. ” 

• If the teachers are exceptional in their ability to use [specific enhancement] strategies 
for managing conflicts, as part of classroom procedures, or to help build positive rela- 
tionships between the children, select “5 = strongly agree. ” 

5. The teachers model and actively promote [specific enhancement] strategies. 

• If teachers’ behaviors are inconsistent with, or undermine proper use of [specific en- 
hancement] strategies, select “1 = strongly disagree. ” 

• If the teachers rarely model proper use of [specific enhancement] strategies, select “2. ’’ 

• If the teachers occasionally model proper use of [specific enhancement] strategies, but 
not consistently, select “3. ” 
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• If the teachers often model and actively promote [specific enhancement] strategies, se- 
lect “4. ” 

• If the teachers are exceptional in their ability to model and actively promote [specific 
enhancement] strategies, select “5 = strongly agree. ” 


Fidelity of Teaching and Supporting Children in [Specific Enhancement] 

1. The teachers are prepared for [specific enhancement] activities and seem familiar with what 

to do. 

• If the teachers are never prepared or familiar with [specific enhancement] activities, 
select “1 = strongly disagree. ” 

• If the teachers only rarely are prepared or familiar with [specific enhancement] activi- 
ties, select “2. ” 

• If the teachers occasionally are prepared or familiar with [specific enhancement] activi- 
ties, but not consistently, select “3. ” 

• If the teachers are usually prepared or familiar with [specific enhancement] activities, 
select “4. ” 

• If the teachers are exceptional in their preparation or familiarity with [specific en- 
hancement] activities, select “5 = strongly agree. ” 

2. The teachers use many of the [specific enhancement] strategies, and modifications or 

additions are consistent with the [specific enhancement] goals and objectives. 

• If the teachers never use [specific enhancement] strategies, select “1 = strongly disa- 
gree. ” 

• If the teachers rarely model or promote [specific enhancement] strategies, and any mod- 
ifications or additions are inconsistent with [specific enhancement] goals and objec- 
tives, select “2. ” 

• If the teachers occasionally model or promote [specific enhancement] strategies, but not 
consistently, and any modifications are usually inconsistent with [specific enhance- 
ment] goals and objectives, select “3. ” 

• If the teachers often model and actively promote [specific enhancement] strategies, and 
any modifications are usually consistent with [specific enhancement] goals and objec- 
tives, select “4. ” 

• If the teachers are exceptional in their ability to model and actively promote [specific 
enhancement], and modifications are always consistent with [specific enhancement] 
goals and objectives, select “5 = strongly disagree. ” 
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3. Material is presented in an engaging manner. The teachers are positive, energetic and 
enthusiastic about [specific enhancement]. There is flexibility in the presentation and the 
teachers appear comfortable with [specific enhancement]. 

• If the teachers either never do [specific enhancement] activities, or are very hostile to- 
wards [specific enhancement], never presenting it an engaging manner and are always 
very unenthusiastic about [specific enhancement], select “1 = strongly disagree. ” 

• If the teachers are rarely positive, energetic, or enthusiastic about [specific enhance- 
ment], and often appear uncomfortable with [specific enhancement], select “2. ’’ 

• If the teachers occasionally present [specific enhancement] in an engaging manner, but 
not consistently, select “3. ” 

• If the teachers usually present the material in an engaging manner, and are positive, en- 
ergetic, and enthusiastic about [specific enhancement], select “4. ” 

• If the teachers are exceptional in their ability to present materials, and in their energy 
and enthusiasm about [specific enhancement], select “5 = strongly agree. ” 

4. The teachers are patient and sensitive to the skill level of the children and adapt their style 
of presentation and pacing to match the children. 

• If the teachers are never patient or sensitive to the skill level of the children, select “1 = 
strongly disagree. ” 

• If the teachers are rarely patient or sensitive to the skill level of the children, select “2. ” 

• If the teachers are occasionally patient or sensitive to the skill level of the children, but 
not consistently, select “3. ” 

• If the teachers are usually patient and sensitive to the skill level of the children, select 
"4. ” 

• If the teachers are exceptional in their ability to adapt their style of presentation to 
match the skill level of the children, select “5 = strongly agree. ” 

5. The children are responsive to [specific enhancement] strategies; the [specific enhance- 
ment] strategies are effective in this classroom. 

• If teachers are not using any [specific enhancement] strategies, select “1 = strongly dis- 
agree. ” 

• If most of the children are rarely responsive to [specific enhancement] strategies, and 
the [specific enhancement] strategies are not being effective, select “2. ” 

• If most of the children are occasionally responsive to [specific enhancement] strategies, 
but not consistently, select “3. ” 

• If most of the children are responsive to [specific enhancement] strategies, and they are 
effective in the classroom, select “4. ” 

• If most of the children are very responsive to [specific enhancement] strategies, and the 
strategies are very effective in the classroom, select “5 = strongly agree. ” 
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Coach Monthly Fidelity Report: Enhancement-Specific 

Fidelity Items 


Fidelity of Programmatic Activities (Lead Teacher): Incredible Years (lY) 

1. Building relationships with students (warm, fun, positive greetings, one-on-one time, eye 
level, special interest activities, compliment circle time). 

• If there is no evidence that the teacher is making efforts to build personal relationships 
with students, or its use is flawed, select “1 = strongly disagree. ” 

• If the teacher only engages in relationship building activities a few times a week, select 
“ 2 . ” 

• If the teacher engages in relationship building activities a few times a day, select “3. ’’ 

• If the teacher consistently works on relationship building in interactions with students 
throughout most of the day, select “4. ” 

• If the teacher is exemplary in his or her relationship-building interactions with students, 
select “5 = strongly agree. ” 

2. Use of proactive teaching strategies (clear classroom mles, “show me five” poster and cue 
cards, clear commands, predictable transitions, when-then statements, posted schedule). 

3. Building relationships with parents (telephone calls home, weekly newsletters, lY teacher- 
parent activities sent home and recommended for each workshop, parent input into behavior 
plans, invites parents to classroom). 

4. Behavior plans (implement behavior plans and specific strategies). 

5. Praise and encouragement (labeled, specific praise, proximal praise, praise “positive 
opposites,” praise to other teachers). 

6. Academic and persistence coaching (avoid use of questions, criticisms and commands, 
describe academic concepts and child’s state and efforts). 

7. Social coaching (target social behaviors to increase, model social skills in one-on-one 
interactions with students, label social behaviors when they occur). 

8. Emotion and self-regulation coaching (target positive opposite emotions, build emotional 
literacy language, teach calm-down practices). 
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9. Motivating children with incentives (for example, spontaneous surprises, celebrations, 
announcements, cheers, positive notes home, special privileges for rewards, sticker 
systems). 

10. Ignoring, distractions, and redirects (for example, start with least intmsive, follow behavior 
plan, target behaviors to ignore, proximity praise, create opportunities to coach and practice, 
distractions, praise positive opposites, teach other children to ignore, redirect, calming self- 
talk). 

1 1 . Consequences and time out to calm down (teach children time-out to calm down, track use 
of time-out and revisit behavior plan, use prompts and reminders, use “if-then” statements, 
use time out only for aggressive behavior, provide opportunities to practice desired behav- 
ior, calm-down self-talk). 

12. Teaching problem-solving skills (teach problem solving steps in circle time and implement 
when conflict occurs). 

13. Supporting and praising co-teachers/other adults and using positive coping strategies (create 
supportive climate with other adults in classroom, use self-praise, positive self-talk, and per- 
sonal time outs when needed, incorporate self-care goals). 
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Fidelity of Programmatic Activities (Lead Teacher): Preschool PATHS 


1 . PATHS lessons 

• If there is no evidence that the teacher is implementing a regular PATHS lesson or its 
use is significantly flawed (for example, unprepared, digression from lesson content or 
poor understanding of concepts, inappropriate pacing, failure to engage students or as- 
sess comprehension during lesson), select “1 = strongly disagree. ” 

• If PATHS lessons are delivered infrequently or if there are problems with lesson deliv- 
ery (for example, unprepared, digression from lesson content or poor understanding of 
concepts, inappropriate pacing, failure to engage students or assess comprehension dur- 
ing lesson), select “2. ” 

• If PATHS lessons are implemented regularly but there are some problems (for example, 
unprepared, digression from lesson content or poor understanding of concepts, inappro- 
priate pacing, failure to engage students or assess comprehension during lesson), select 
“S. ” 

• If the teacher implements the PATHS lesson weekly and the majority of the time, the 
lessons are delivered well (for example, prepared, lesson content covered, understand- 
ing of concepts, appropriate pacing, engages students and assesses student comprehen- 
sion during lesson), select “4. ” 

• If the teacher implements a PATHS lesson weekly, and is exemplary in his or her deliv- 
ery (for example, prepared, lesson content covered, understanding of concepts, appro- 
priate pacing, engages students and assesses student comprehension during lesson), se- 
lect “5 = strongly agree. ” 

2. PATHS Kid of the Day (PKD). 

3. Building a caring classroom: Classroom structure. 

4. Building a caring classroom: Teacher-student relationships. 

5. Nurturing emotional understanding: Emotional modeling. 

6. Nurturing emotional understanding: Emotion expression and support. 

7. Fostering self-regulation: Turtle Technique. 

8. Fostering self-regulation: FREE strategies. 

9. Supporting problem solving. 

10. Discipline. 
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Fidelity of Programmatic Activities (Lead Teacher): Tools of the Mind 

1 . Play planning and scaffolded writing. 

• If either play planning is not being implemented at all or so infrequently as to have little 
to no impact on the students (once a week), select “1 = strongly disagree. ” 

• If play planning is being implemented on a regular basis (every day with most to all of 
the children), but the teacher is still stmggling with the organization and flow of the ac- 
tivity (for example, the activity may take too long or children may be allowed to scrib- 
ble and then leave the table), select “2. ’’ 

• If play planning is implemented on a regular basis, and the teacher has a firm grasp on 
the sfructure and flow of the activity (wheel is ready, role cards are used, plans are re- 
viewed from the day before) although there may be minor errors made (for example, 
scaffolding around the table rather than going to the student who needs the most assis- 
tance first) and the teacher still stmggles with scaffolding within the ZPD (zone of prox- 
imal development) and tends to provide similar instruction to all (for example, either 
under or overscaffolding by working on picture or lines with everyone even though 
some students might be ready for more advanced skills), select “3. ” 

• If play planning is implemented on a regular basis and the stmcture and flow of the ac- 
tivity has been completely internalized by the teacher, and the teacher is able to recog- 
nize different ZPDs of students and adjust her insfruction accordingly some to most of 
the time, select “4. ” 

• If the teacher is fluent in play planning, moving with ease from student to student, and a 
range of scaffolding can be seen during the activity with students working at all levels 
(for example, some focused on “making a plan” and drawing themselves while others 
might be writing ending sounds in words); and the teacher encourages children to talk 
and work together (that is, buddy planning) and potential conflicts are worked out at the 
table before children leave to play, select “5 = strongly agree. ” 

2. Make-believe play (MBP). 

3. Make-believe play practice (MBPP). 

4. Attention-gathering activities (for example, fmgerplays, make-believe transitions). 

5. Graphics practice/buddy reading. 

6. Overall schedule of the day (short group times, transitions used, no long waiting times, 

percentage of day filled with Tools activities). 
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The implementation data sources used in this report are listed below, by the reporter of the data. 


Data from Trainers 

Trainer Fidelity Logs (MIS) 

Trainers visited classrooms two to three times a year: once at the beginning of the school year, 
and once in the second half of the school year (January through May). After those visits, trainers 
used the management information system (MIS) to rate the teaching team on modeling and 
generalization, fidelity of teaching and supporting children, and fidelity of programmatic 
activities. 

Global Measure of Fidelity 

Trainers reported on “modeling and generalization” in the classroom and the “fidelity of 
teaching and supporting children in [specific enhancement].” The 10 items in the two combined 
sections were answered using a 1 (have considerable difficulty/strongly disagree) to 5 (highly 
skilled/strongly agree) scale.' The 10 items were averaged together at each time point. Internal 
consistency was high (Cronbach’s atrainers= 0.97). 

Average fidelity score. For each classroom, trainer fidelity ratings were averaged across 
the year to create an average trainer rating. The trainer ratings were averaged together with 
coach average ratings (described below) to obtain an overall fidelity score for each classroom. 

Trainer Logs of Coach Quality (MIS) 

After their visits, trainers also used the MIS to rate coach quality. Trainers rated the 
coaches’ skills, knowledge of the enhancement, and ability to support the teachers. Trainers 
responded to 10 questions on a scale of 1 (strongly disagree) to 5 (strongly agree). Questions 
address whether, for example, “the coach ably demonstrates techniques or strategies of pro- 
gram” and “the coach motivates his or her teachers.” The items were averaged at each time 
point and then across time points; internal consistency was high (Cronbach’s a= 0.96). 

Trainer Supervision Logs (MIS) 

Trainers and coaches had a schedule of supervisory phone calls and in-person visits 
throughout the year, which varied by enhancement. Each time that they were in contact with 

'in Cohort One (program year 2009 to 2010), coaches and trainers answered questions on a scale of 1 
(have considerable difficulty) to 5 (highly skilled), while in Cohort Two (program year 2010 to 2011), coaches 
and trainers answered questions on a scale of 1 (strongly disagree) to 5 (strongly agree). The items were 
worded slightly differently in each cohort to match the wording of the responses. 
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coaches, trainers used the MIS to report on the type and duration of their communication with 
coaches, and the overall content of the interactions. 

Trainer Interviews 

In spring of the implementation year, trainers were interviewed on the phone by the re- 
search team (MDRC and MEF Associates) about each grantee with which they worked. 
Trainers reflected on how implementation was handled by the teachers and in the classrooms 
that year, their relationship and sessions with the coach, and their interactions with the develop- 
ers of the intervention. The interviews featured semi-stmctured questions that were created to be 
parallel across interview respondents (for example, trainers, coaches, and teachers), with 
specific subsets of questions selected for each type of interview. The parallel themes were 
background information, training, program model implementation, the trainer-coach relation- 
ship, coaching sessions, the teacher dyad, informal peer coaching, organizational setting, 
trainer-coach sessions, sustainability, the developer-trainer relationship, and summary reflec- 
tions. The interview analysis process is detailed under Additional Information on Qualitative 
Data Analysis, at the end of this appendix. 


Data from Coaches 

Coach Demographics Survey 

Coaches responded to 15 questions that asked about basic demographics such as their 
age, race, ethnicity, education, and relevant background. 

Coach Monthly Fidelity Logs (MIS) 

At the end of each month, coaches used the MIS to enter reflections on their assigned 
teachers and classrooms. They reported on the teachers’ response to enhancement-specific 
coaching, consultation, and implementation; modeling and generalization of enhancement- 
specific practices throughout the whole school day; and fidelity of teaching and supporting 
children. Appendix B describes the items in the two sections about modeling and generalization 
and fidelity of classroom implementation. They also reported on the teachers’ fidelity of 
classroom implementation, as shown in Appendix B. Finally, they scored the organizational 
support they received and the coteacher relationship. 

Global Measure of Fidelity 

All global fidelity variables (described below) were created using sections of the logs in 
which coaches reported on “modeling and generalization’’ in the classroom and the “fidelity of 
teaching and supporting children in [specific enhancement].’’ As with the trainer global measure 
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of fidelity, the 10 items in the two combined sections were answered using the scale of 1 (have 
considerable difficulty/strongly disagree) to 5 (highly skilled/strongly agree), ^ as described in 
Appendix B. The 10 items were averaged together at each time point. Internal consistency was 
high (Cronbach’s acoaches= 0-97). 

Average fidelity score. For each classroom, coach monthly fidelity average scores were 
averaged across the year to create an average coach rating of fidelity. The coach and trainer 
ratings were averaged together to obtain an overall fidelity score for each classroom. 

Start-of-year score. Each classroom’s September fidelity average from the coach month- 
ly fidelity log was used as the start-of-year score. If the September coach monthly fidelity log 
was not submitted, the October fidelity average was used instead. If the October coach monthly 
fidelity log was not submitted for a classroom, the start-of-year score was considered missing. 

End-of-year score. Each classroom’s April fidelity average from the coach monthly fi- 
delity log was used as the end-of-year score. If the April coach monthly fidelity log was not 
submitted, the March fidelity average was used instead. If the March coach monthly fidelity log 
was not submitted for a classroom, the end-of-year score was considered missing. 

Change score. The change score for each classroom was calculated as the difference 
between the end-of-year score and start-of-year score, as described above. 

Enhancement-Specific Measure of Fidelity 

Fidelity to the core components of each enhancement by lead and assistant teachers was 
assessed through enhancement-specific questions about teachers’ implementation of developer- 
identified core components of the three enhancements. The section consisted of between 6 and 
13 items, depending on the enhancement; all were answered using the scale of 1 (have consid- 
erable difficulty/strongly disagree) to 5 (highly skilled/strongly agree) that is described in 
Appendix B. Items varied slightly across cohorts, and were matched where feasible.^ Responses 
to each question were averaged across the year."^ 


^In Cohort One (program year 2009 to 2010), coaches and trainers answered questions on a scale of 1 
(have considerable difficulty) to 5 (highly skilled), while in Cohort Two (program year 2010 to 2011), coaches 
and trainers answered questions on a scale of 1 (strongly disagree) to 5 (strongly agree). The items were 
worded slightly differently in each cohort to match the wording of the responses. 

^In Cohort One (program year 2009 to 2010), coaches and trainers reported on the teachers in the class- 
room, providing a single score for each classroom. In Cohort Two (program year 2010 to 2011), coaches and 
trainers gave one score to the lead teacher and one score to the assistant teacher. The lead and assistant teacher 
data from Cohort Two were averaged to match with the Cohort One data. The number of items depended on 
the enhancement and the cohort. Only those items that were similar across cohorts were used. In some cases, 
two items from one cohort were averaged to match up with a single item in the other cohort. 

frn Cohort Two, the questions were first averaged across lead and assistant teachers each month. 
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Coach Weekly Logs (MIS) 

At the end of each week, coaches reported through the MIS on their weekly meeting 
and observation of teachers, as shown in Appendix B. The coach weekly log consisted of six 
questions in total, which covered (1) time spent with the teacher since the last coach session; (2) 
time spent in the classroom coaching, modeling, or observing since the last coaching session; 
(3) time spent preparing for coaching; (4) extent of teachers’ use of enhancement practices, 
lessons, and activities; (5) strategies that coaches used during their meeting with teachers (such 
as answering questions about the program strategies, role playing, setting goals, and so forth); 
and (6) how productive the meeting was. Frequency and dosage data were averaged across 
enhancement and across month to determine the overall dosage of coaching, the dosage for each 
enhancement, and the dosage for each month. 

Coach End-of-Year Reflections 

At the end of the year, coaches in Cohort Two reflected on their experience participat- 
ing in Head Start CARES. They reported on their practices and beliefs about coaching and 
supervision, as well as the facilitators of and challenges to successful coaching. The coach end- 
of-year reflections consisted of five sections, which focused on background information, social- 
emotional beliefs, coaches’ coaching experience in Head Start CARES, coaches’ experience 
with supervisors and mentors, and final thoughts. Coaches reflected on items such as, “In your 
opinion, how effectively does your enhancement help teachers manage their classrooms?’’ and 
“In general, how much time do you think you need to spend each week on ONE classroom?’’ 
Coaches also reflected on, for instance, “What characteristics or skills are most important for 
coaches to have in order to best support classroom implementation?’’ 

Coach Training Feedback Forms 

After the first and last training of each enhancement, coaches were asked to rate the 
training. These feedback forms captured participant reactions to training content, methods, 
dynamics/interactions, and participant perceptions of the overall quality of each training 
workshop. The training feedback form asked coaches to respond to 1 1 questions that were rated 
on a scale of 1 (strongly disagree) to 5 (strongly agree). Questions sought to determine whether, 
for example, “The material was presented in an accessible and easy-to-understand manner’’ and 
“I would recommend this workshop to teachers.’’^ 


^“Workshop” in this appendix refers to a training session. 
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Coach Calls with the Technical Assistance Team 

For Cohort Two, the technical assistance team facilitated coach calls, which allowed 
coaches around the country from the same enhancement to talk about their experiences in the 
classroom. Two calls per enhancement were scheduled for each time point, with between five 
and eight coaches participating in each call, along with trainers when possible. Coaches were 
asked to reflect on various aspects of coaching, such as building relationships, coaching strate- 
gies, and lead and assistant teacher relationships. Coaches also used the call as an opportunity to 
raise challenges and share successful strategies for overcoming those challenges. The technical 
assistance team took notes during the calls, which were analyzed to help identify qualitative 
themes. 


Coach Interviews 

All of the 52 Head Start CARES coaches were interviewed by the research team during 
the spring of the implementation year, and all 52 interviews were coded. The majority of coach 
interviews were in person, although some took place on the phone when the coach was not 
available at the time of the site visit. Coaches reported on their impressions of the enhancement, 
training, program model implementation, the trainer-coach relationship, coaching sessions, the 
teacher dyad (that is, the lead and assistant teacher pairs), informal peer coaching, and the 
organizational setting, as well as trainer-coach sessions. As with trainer interviews, the inter- 
views featured semistmctured questions that were created to be parallel across respondents, with 
specific subsets of questions designed for each type of interview. The parallel themes were 
background information, impressions of the enhancement, training, program model implemen- 
tation, the trainer-coach relationship, coaching sessions, the teacher dyad, informal peer coach- 
ing, organizational setting, trainer-coach sessions, sustainability, and summary reflections. The 
interview analysis process is detailed under Additional Information on Qualitative Analysis at 
the end of this appendix. 


Data from Teachers and Classrooms 

Data from Teachers 

Lead Teacher Self-Survey (Baseline and Follow-Up) 

The lead teacher self-survey was collected in the spring of the year before classrooms 
implemented the Head Start CARES enhancements and in the spring of the implementation 
year. The lead teacher self-survey (baseline) included nine sections: (1) lead teachers’ de- 
mographics (including age, race, education, other credentialing, and financial and family 
situation); (2) information about the classroom; (3) emotion coaching (adapted Emotion-Related 
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Parenting Styles Self-Test, short version, emotion coaching subscale); *’ (4) burnout (Maslach 
Burnout Inventory emotional exhaustion subscale, known as MBI); ^ (5) psychological distress 
(K-6 Kessler Psychological Distress Scale);* (6) adaptability to change (Texas Christian 
University Organizational Readiness to Change); (7) views on social-emotional development; 
(8) previous training and professional development; and (9) the lead-assistant teacher relation- 
ship. The lead teacher self-survey (follow-up) asked program teachers to report on implementa- 
tion-specific items. The follow-up survey included four additional sections: (10) implementa- 
tion of social-emotional practices, (11) the coach-teacher relationship, (12) motivation to 
implement, and (13) teachers’ relationship with their supervisor. 

In the time between baseline in the spring and the start of Head Start CARES imple- 
mentation in the fall, some teachers left the study. The research team attempted to match the 
teachers who answered the survey at baseline to the teachers who implemented the enhance- 
ment in the fall to understand how much turnover occurred and to make determinations about 
missing data. 

Key scales covered in the lead teacher self-survey are detailed below. 

Emotion coaching. Emotion coaching was defined as teachers’ ability to positively sup- 
port children’s navigation of negative or difficult emotions. The emotion coaching subscale 
comprises four items from the short version of the Emotion-Related Parenting Styles Self- 
Test.'® Lead teachers responded to questions on the lead teacher self-survey such as, “When a 
child in my classroom is sad, we sit down to talk over the sadness,” and “When a child in my 
classroom gets angry, it is important to help the child find out what caused his/her anger.” A 
factor analysis was conducted on the scale as it appeared in Cohort One. A score was created by 
averaging the four items; “When my child gets angry, it’s time to solve a problem,” was 
dropped from the subscale. Lead teachers in Head Start CARES rated themselves as generally 
extremely supportive of children’s displays of negative emotion, reporting a 3.56 on a scale 
from 0 (low) to 4 (high). 

MBI emotional exhaustion subscale. Lead teachers’ ratings of emotional exhaustion 
and overextension at work was assessed using the MBI educator rating scale of 0 (low) to 54 
(high). Teachers rated nine items on a scale of 0 to 6. Items were added to create a total score 
with the mean of non-missing values imputed for missing values. In general, the Head Start 
CARES sample reported low mean levels (a score of around 14), while the MBI validation 

®Hakim-Larson et al. (2006). 

^Maslach, Jackson, and Leiter (1996). 

"Kessler etal. (2003). 

^Lehman, Greener, and Simpson (2002). 

*®Hakim-Larson et al. (2006). 

"Maslach and Jackson (1981). 
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sample, which included over 450 teachers from Massachusetts, reported mean levels of emo- 
tional exhaustion around a score of 21.'^ 

Kessler Psychological Distress Scale (K-6). The Kessler Psychological Distress Scale 
(K-6) includes six questions that ask teachers about their emotional state and are scored on a 
scale of 0 (none of the time) to 4 (all of the time). Scores from the six questions are summed 
with the mean of non-missing values imputed for missing values, producing a score between 0 
and 24. Low scores indicate low levels of psychological distress, and high scores indicate high 
levels of psychological distress. Questions include, for example, “During the last 30 days, about 
how often did you feel nervous?” and “During the last 30 days, about how often did you feel 
worthless?” Lead teachers in Head Start CARES generally rated themselves as having low 
distress, reporting a 3 on a scale from 0 to 24. 

Adaptability to change. The Adaptability subscale of the TCU Organizational Readi- 
ness to Change instmment, collected at baseline, is made up of four items, including, “You are 
willing to try new ideas even if some people are reluctant,” “Learning and using new proce- 
dures are easy for you,” “You are able to adapt quickly when you have to shift focus,” and “It is 
easy to change routine procedures to meet new conditions.” Given that this scale was previously 
used in clinical settings instead of with teachers, additional psychometric work, including an 
exploratory factor analysis, was completed. The items were averaged together to create a 
composite score; internal consistency was adequate (Cronbach’s a = 0.71). In general, teachers 
rated themselves highly on adaptability, averaging a score of 3 on a scale from 0 (low) to 4 
(high) on their ease with and openness to trying new things. 

Views on social-emotional development. Teachers responded to a question asking what 
they valued more: children’s academic readiness or social-emotional readiness. Teachers were 
able to select that they valued “academic readiness” or “social-emotional readiness” more, with 
the middle considered neutral. Most teachers (78 percent) rated themselves as neutral. 

Lead teacher and assistant teacher relationship. Lead teacher and assistant teacher rela- 
tionship was based on two questions about each lead teacher’s perception of her relationship 
with the assistant teacher. The lead teacher rated the frequency of tmst and collaborative work 
in the relationship on a scale of 1 (never) to 5 (always). The measure of the lead and assistant 
teacher relationship was the average of these two items. Lead teachers rated the relationship 
highly, with an average score of 4.53. 


*^For more information, see Maslach, Jackson, and Leiter (1996). 
'^Kessler et al. (2002). 

'"'Lehman, Greener, and Simpson (2002). 
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Motivation to Implement the Enhancement 

The wording varied slightly by cohort and enhancement, but motivation to implement 
was assessed with one item: “I feel motivated to use the strategies/program/approach in my 
classroom.” Teachers rated their motivation on a scale of 1 (strongly disagree) to 5 (strongly 
agree). In Cohort One, data on teachers’ motivation to implement the enhancement was collect- 
ed from lead and assistant teachers after each training session. In Cohort Two, data on teachers’ 
motivation to implement the enhancement was collected from teachers between late September 
and November of the implementation year.'^ Unless it was missing, to create a fall score, the 
earliest rating from September to November was used for Cohort One; the teachers’ fall rating 
was used for Cohort Two. At the start of the year, teachers rated themselves, on average, as 
highly motivated to implement their enhancement, with an average score of 4.18. 

Teacher Training Feedback Forms 

After the first and last training session for each enhancement, teachers were asked to 
rate the training. These feedback forms captured participant reactions to training content, 
methods, dynamics/interactions, and participant perceptions of the overall quality of each 
training session. Lead and assistant teachers were asked to respond to six questions on a scale of 
1 (strongly disagree) to 5 (strongly agree). Many of the questions were identical to the questions 
that coaches answered, including, “The material was presented in an accessible and easy-to- 
understand manner” and “Information presented by the trainer was useful.” There were also 
some teacher-specific questions, such as, “The training prepared me to successfully implement 
the program’s strategies with my students.” 

Lead Teacher and Assistant Teacher Interviews 

A total of 126 interviews with program lead teachers and 80 interviews with program 
assistant teachers were conducted. Lead teacher interviews were generally conducted individu- 
ally, while assistant teachers were often interviewed in a group. In each grantee, three centers, 
each implementing a different enhancement, were selected for interviews. The interview team 
interviewed as many lead and assistant teachers in person in each selected center as time 
allowed. Lead and assistant teachers reflected on their impressions of program models, training, 
program model implementation, coaching sessions, the teacher dyad, informal peer coaching, 
and the organizational setting. As with coaches and trainers, the interviews featured semistmc- 
tured questions that were created to be parallel across respondents, with specific subsets of 
questions designed for each type of interview. The parallel themes were background infor- 
mation, impressions of the enhancement, training, program model implementation, the trainer- 

*^In Cohort Two (program year 2010 to 2011), only one teacher per classroom, usually the lead teacher, 
completed an assessment of motivation to implement. 
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coach relationship, coaching sessions, the teacher dyad, informal peer coaching, organizational 
setting, sessions with the trainer and coach, sustainability, relationship between the developer 
and the trainer, and summary reflections. 

Despite early research suggesting that opinion leaders and peer mentors (teachers who 
mentor other teachers) in education settings were able to bolster teachers’ implementation of a 
set of teaching practices,'*’ Head Start CARES teachers reported little informal peer support or 
contact with other teachers about implementation at all. 

As detailed below, a subset of the teacher interviews (80 lead teacher interviews and 49 
assistant teacher interviews) were selected for analysis. The interview analysis process is 
detailed under Additional Information on Qualitative Data Analysis at the end of this appendix. 

Lead teachers. Interview teams were asked to select two teachers from classrooms with 
high fidelity scores and two teachers from classrooms with low fidelity scores from each 
grantee for analysis, with the goal of providing a wide range of implementation experiences. 
Regardless of fidelity rating, all interview teams were asked to prioritize interviews in which 
teachers were able to clearly articulate their feelings and provide insight into their experience 
participating in Head Start CARES. In situations where the interview team had trouble distin- 
guishing which teachers fell into the high and low categories, they were asked to choose the 
overall “best” interviews. 

Assistant teachers. Assistant teacher interviews sometimes occurred in a group setting. 
The interview teams were asked to select two high-fidelity assistant teacher interviews (or 
group interviews) from each grantee for analysis. Regardless of fidelity rating, all interview 
teams were asked to prioritize interviews in which assistant teachers were able to clearly 
articulate their feelings and provide insight into their experience participating in Head Start 
CARES. 

Teacher Training Attendance Forms 

At each training, coaches collected attendance forms to document which teachers were 
at the training. 


'^Atkins, Graczyk, Franzier, and Abdul-Adil (2003). 
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Data from Classrooms 


Baseline Scores from the Classroom Assessment Scoring System (CLASS) — 
Preschool Version 

In the spring before implementation, observers who were blind to the intervention status 
of the classrooms observed all adults (including both teachers) in the classroom for one day. 
CLASS is a nationally used measure of classroom quality.'^ It provides global, seven-point 
Likert scores in four aspects of the classroom climate, as shown in Appendix Table C.l: 
(1) emotional support, which includes positive climate, negative climate, teacher sensitivity, and 
regard for student perspectives; (2) classroom organization, which includes behavior manage- 
ment, productivity, and instmctional learning formats; (3) instructional support, which includes 
concept development, quality of feedback, and language modeling; and (4) literacy focus. Like 
the lead teacher self-survey data, teachers who were present for the baseline classroom observa- 
tions were not always the same teachers who implemented the enhancement the following 
school year. CLASS is coded in four segments; each segment consists of 20 minutes of observa- 
tion followed by 10 minutes of coding. 

CLASS scores. The score for each of the 10 dimensions that make up the emotional 
support, classroom organization, and instmctional support domains was calculated as the 
average of the scores on that dimension across the four segments. The score for each of the 
three domains noted above was calculated as the average of the scores of its dimensions. The 
division of items into factors was determined by confirmatory factor analysis, based on the 
widely used Pianta 3-factor solution.'* Confirmatory factor analysis led to selection of the 
Pianta 3 -factor solution. 

Adapted Teaching Style Rating Scale (Adapted TSRS) 

In the spring before implementation and in the spring of the implementation year, ob- 
servers who were blind to the intervention status of the classrooms observed the lead teacher in 
the classroom for one day. The Adapted TSRS was adapted for the Head Start CARES project 
by Dr. Cybele Raver from the original TSRS measure, used in the REDI study. The Adapted 
TSRS was created to measure the core components of each of the three enhancements as they 
were implemented effectively in the classroom. Appendix F shows the Adapted TSRS as it was 
used in Head Start CARES. Teachers were rated on a five-point Likert scale on three teacher 


'^La Paro, Pianta, and Stuhlman (2004). 

'*La Paro, Pianta, and Stuhlman (2004). 

'^Domitrovich, Cortes, and Greenberg (2000). 

^"REDI stands for REsearch-based, Developmentally Informed. 
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Head Start CARES Demonstration 


Appendix Table C.l 


CLASS Factor Analysis Structure 


Dimension 

Emotional 

Support 

Classroom 

Organization 

Instructional 

Support 

Positive climate 

X 





Negative climate 

X 

- 

- 

Teacher sensitivity 

X 

- 

- 

Regard for student perspectives 

X 

- 

- 

Behavior management 

- 

X 

- 

Productivity 

- 

X 

- 

Instructional learning formats 

- 

X 

- 

Concept development 

- 

- 

X 

Quality of feedback 

- 

- 

X 

Language modeling 

- 

- 

X 


SOURCE: MDRC calculations based on observational assessments completed using 
the Classroom Assessment Scoring System (CLASS); see Pianta, LaParo, and Hamre 
( 2008 ). 


practices, as shown in Appendix Table C.2: (1) classroom management, which includes 
consistency/routine, preparedness, classroom awareness, positive behavior management, 
negative behavior management, and attention/engagement; (2) social-emotional instruction, 
which includes emotion modeling, emotion expression, emotion regulation, social awareness, 
social problem-solving, and provision of interpersonal support; and (3) scaffolding, which 
includes scaffolding dramatic play and scaffolding peer interaction. The Adapted TSRS is 
coded in two segments at the same time that CLASS is coded. Each Adapted TSRS segment is 
made up of 40 minutes of observation followed by 10 minutes of coding. 

Adapted TSRS scores. The division of items into factors was determined by exploratory 
factor analysis. Based on scree plots and eigenvalue tables, both the Cohort One and Cohort 
Two data imply a three-factor solution, with eigenvalues above a 1 for all three factors. The 
items generally aligned with the theoretical factor stmcture, as shown in Appendix Table C.2. 
On the scaffolding variable, a third item. Talk Aloud, was dropped because of a low and 
nontheoretically justified loading (0.34) and low correlations with other items. 
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Data from Administrators/Grantee 


Site Team Discussions with the Grantee 

Throughout the course of the study, site teams held monthly phone calls with grantee li- 
aisons. Site teams comprised operations staff from MDRC and MEF Associates. During site 
recmitment and these calls, site teams asked grantees about aspects of the organization, includ- 
ing stmctural and procedural issues related to the project. While questions varied by site, some 
of the questions that were asked of administrative staff included, for example, “What curricula 
are in place at your sites?”; “What other assessment tools are being used at your sites?”; and 
“When are teachers expected to complete these assessments (once a year, twice a year, and so 
on)?” The site teams took notes during their check-ins, which were incorporated into research 
team summary impressions about each grantee. 


Head Start CARES Demonstration 
Appendix Table C.2 

Adapted TSRS Factor Analysis Structure 


Dimension 

Classroom 

Management 

Social-Emotional 

Instruction 

Scaffolding 

Consi stency/ routine 

X 

_ 

_ 

Preparedness 

X 

- 

- 

Classroom awareness 

X 

- 

- 

Positive behavior management 

X 

- 

- 

Negative behavior management 

X 

- 

- 

Attention/engagement 

X 

- 

- 

Emotion modeling 

- 

X 

- 

Emotion expression 

- 

X 

- 

Emotion regulation 

- 

X 

- 

Social awareness 

- 

X 

- 

Social problem-solving 

- 

X 

- 

Interpersonal support 

- 

X 

- 

Scaffolding dramatic play 

- 

- 

X 

Scaffolding peer instruction 

- 

- 

X 

Talk aloud 

— 

— 

— 


SOURCE: MDRC calculations based on observational assessments completed using the 
Adapted Teaching Style Rating Scale (Adapted TSRS). 
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Head Start Center Supervisor Interviews 

Center supervisors were asked about their impressions of the program model training, 
program model implementation, coaching sessions, the organizational setting, summary 
reflections, and sustainability. The interviews featured semistmctured questions that were 
created to be parallel across respondents, with specific subsets of questions designed for each 
type of interview. The parallel themes were background information, impressions of the 
program models, training, program model implementation, trainer-coach relationship, coaching 
sessions, teacher dyad, informal peer coaching, organizational setting, sustainability, the 
developer-trainer relationship, and summary reflections. 

Grantee Liaison Interviews 

The 17 grantee liaisons were interviewed during the spring of the implementation year. 
The grantee liaisons explained their impressions of the program models and program model 
implementation, and they also described coaching sessions, the organizational setting, summary 
reflections, and sustainability. The interviews featured semistmctured questions that were 
created to be parallel across respondents, with specific subsets of questions designed for each 
type of interview. The parallel themes were background information, impressions of the 
program models, training, program model implementation, trainer-coach relationship, coaching 
sessions, teacher dyad, informal peer coaching, organizational setting, summary reflections, 
trainer-coach sessions, sustainability, and the developer-trainer relationship. All sections of the 
grantee liaison interviews were coded, since grantee liaisons were generally involved in the 
implementation process and were able to provide more global assessments of the implementa- 
tion process. Grantee liaisons also worked closely with administrators and were able to provide 
insight on enhancement sustainability and organizational support. 

Data from Developers 

Developer Discussions with the Technical Assistance Team 

Throughout the year, the research team held periodic phone conversations with the de- 
velopers of the enhancements in order to understand how the developers viewed the implemen- 
tation process. These conversations, in addition to a full-day discussion at the end of implemen- 
tation, helped the research team to understand the developers’ perspectives about 
implementation in Head Start CARES. At the end of the implementation year for Cohort Two, 
developers were asked to rank grantees on a low-medium-high scale about their perceptions of 
how training was implemented, how coaching was implemented, how the enhancement was 
implemented in the classroom, and how well the local grantee’s Head Start organizational 
context supported implementation. Developers also reflected on the overall implementation 
process and the lessons learned. 
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Data from Technical Assistance 


Site Visits 

At least one research site visit was completed with each of the 17 Head Start CARES 
grantees. Additionally, some grantees were visited multiple times over the course of the year for 
technical assistance purposes. Control sites were not visited because of funding constraints. 
Interview teams completed descriptions of the organizational context, the environment, and 
other observations they made while visiting the grantee. These site visit descriptions along with 
findings from the interviews informed the research team on contextual factors that affected 
implementation. 

Research Team Training Observation Forms 

From August through January, the research team observed a number of training ses- 
sions across most grantees. The team reported on the trainer’s group process and relationship- 
building skills, leadership skills, methods, knowledge, and the participants’ involvement or 
interest. Members from the research team attended 56 training sessions across the two cohorts, 
observing 9 to 10 sessions per enhancement in Cohort One and 7 to 10 sessions per enhance- 
ment in Cohort Two. Observers responded to questions by noting outstanding, satisfactory, 
poor, or not applicable. Observers were asked to provide additional detail if they answered a 
question with “outstanding” or “poor.” Space was provided at the end of each section for 
additional comments. Examples include, “Created a professional atmosphere,” “Checked for 
understanding throughout delivery,” “Explained rationale for principles covered in a clear, 
convincing manner,” and “Participants’ overall level of participation in the workshop.” 


Additional Information on Qualitative Data Analysis 

Interview Analysis 

Qualitative analysis of interview data was completed using NVivo software. Interview 
teams turned in summary notes of their interviews, which were coded by two coders. In Cohort 
One, a codebook was created to structure how interviews would be coded in NVivo. The 
codebook was modified for Cohort Two. Inter-rater reliability on codes was high (99 percent). 
Below is an overview of how the codebook was created for Cohort One. 

Major Areas of Fidelity and Major Codes 

In the fall of 2010, the Head Start CARES research team identified major areas of in- 
quiry, which included training, coaching, and classroom implementation, as well as documenta- 
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tion of organizational support. These four major areas of inquiry formed the base of the coding 
stmcture in NVivo. 

Sub-codes 

In continuing to expand upon the coding scheme for the Head Start CARES interviews, 
the coders focused on each of the four major areas to identify subcodes that would be included 
under each major code. In order to identify appropriate subcodes, the coders read the interview 
segments within each of the four sections in the Cohort One interviews to identify subcodes that 
would further categorize the information pertaining to the major areas of fidelity. As the coders 
continued to read through the interviews, some of the additional codes and subcodes needed to 
be further specified. Emergent codes were then added to the codebook. For example, “accessi- 
bility,” “time available,” and “organization” were all emergent themes that fit within “organiza- 
tional culture” under “organizational support.” 

Global Implementation Rating Analysis 

At the end of implementation, but before any impact findings were known, the research 
team and the site team held a final meeting to reflect on implementation using information 
collected from trainer fidelity logs, trainer coach quality logs, trainer supervision logs, coach 
monthly fidelity logs, coach weekly logs, trainer interviews, coach interviews, lead and assistant 
teacher interviews, center director interviews, grantee liaison interviews, site visits, and other 
discussions. The team rated each grantee on the quality of training, coaching, classroom 
implementation, and organizational support. The ratings were on a scale of low, medium-low, 
medium, medium-high, and high, and were reached through a process of consensus building. 
This rating took into account all aspects of the implementation process, including training, 
coaching, teacher’s fidelity levels, and the organizational context. In this case, fidelity takes into 
account dosage of training and coaching received, adherence to the training, coaching, and 
enhancement and delivery, as well as the quality of that delivery, while organizational support 
takes into account the support provided within the grantee for implementing the enhancements. 
For example, a high-implementing site might be one in which everyone attended training, 
training was well received, teachers met with coaches weekly and were open to the coaching 
process, and teachers implemented all portions of the enhancements and internalized the content 
as intended by developers. 
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Appendix D 

Cost Estimates for the Head Start CARES Enhancements 
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The cost of the Head Start CARES enhancements consisted primarily of teacher training and 
coaching. The components of these training and coaching costs are described below, and 
estimates of the costs are presented in Appendix Table D. 1 . 

As mentioned in Chapter 2, a technical assistance effort and management information 
system (MIS) were also important to implementing these enhancements. However, the cost 
estimates discussed here include only the costs of the individual enhancements (that is, training 
and coaching for teachers), not the costs of technical assistance and monitoring by MDRC or by 
the enhancement developers, aside from their expenditures on trainers. This technical assistance 
and monitoring effort provided by MDRC was over and above what the models would be likely 
to provide outside of the study; therefore, the costs reported below without technical assistance 
are similar to the costs that a program would encounter. However, this means that additional 
technical assistance or support might be needed to achieve the level of implementation quality 
attained in Head Start CARES. 

Space rental and catering for the training, as well as any cost for training new trainers 
and general consultation provided by trainers, are also not included in the table. The numbers in 
the table are approximations based on proposed budgets and, when possible, invoices, and are 
not exact costs. In addition, these costs are derived from estimates of training costs in 2009- 
2011, which may have changed since the Head Start CARES implementation years. 


Costs in the Context of a Research Demonstration 

The costs of Head Start CARES should also be considered in the context of the national scale of 
the project. Head Start CARES was a national demonstration, so the costs of trainer travel were 
higher in some cases because the trainers had to travel far for the training, while they were 
lower in places where the enhancement already had a bank of trainers. Training was also held at 
regional “hubs,” where training sessions were conducted for teachers from multiple grantees. In 
a nonresearch context, the cost of travel for coaches and teachers, renting a location, and 
catering the training sessions could be lower if they are done on-site. On the other hand, sites 
implementing individually may not be able to share the costs among multiple centers or 
grantees, as some did in Head Start CARES. It is also possible that in a nonresearch context, 
coaching could be provided by existing staff, such as a mental health consultant, instead of an 
outside consultant. 

Head Start CARES was not designed to include a cost study, and implementation costs 
were recorded for accounting purposes only. As a result, costs were totaled by enhancement and 
by grantee, rather than by component of implementation. Appendix Table D.l was created from 
a combination of invoices, which documented actual spending when available, and cost as- 
sumptions created before implementation started. In some cases, these projected costs were 
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Head Start CARES Demonstration 


Appendix Table D.l 
Estimated Program Costs 


Component 

Overall 


Incredible Y ears 

Preschool PATHS 

Tools of the Mind 

Classroom 

Student’’ 

Classroom 

Student 

Classroom 

Student 

Classroom 

Student 

Teacher training 1$) 

5,890 

310 

6,460 

340 

4,670 

250 

6,540 

340 

Trainer fees 

2,760 

150 

2,810 

150 

2,080 

110 

3,410 

180 

Trainer travel costs 

850 

40 

940 

50 

600 

30 

1,030 

50 

Lead and assistant teacher payments 

1,300 

70 

1,560 

80 

1,040 

50 

1,300 

70 

Training materials 

780 

40 

920 

50 

800 

40 

610 

30 

Coach payments 

200 

10 

250 

10 

150 

10 

200 

10 

Coaching ($) 

4,200 

220 

4,530 

240 

4,140 

220 

3,940 

210 

Coach salaries 

3,230 

170 

3,270 

170 

3,160 

170 

3,250 

170 

Coach training 

690 

40 

620 

30 

750 

40 

700 

40 

Coach costs 

410 

20 

430 

20 

400 

20 

400 

20 

Trainer costs 

280 

10 

190 

10 

360 

20 

300 

20 

Trainer supervision'’ 

290 

20 

630 

30 

220 

10 

— 

— 

Total expenditures for all components ($) 

10,090 

530 

10,990 

580 

8,810 

460 

10,480 

550 


SOURCE: MDRC calculations based on fiscal and budget data. 

NOTES: The costs presented in this table are approximations based on both invoices and projected costs, combining those from Cohort One (accmed in 
2009-2010) and Cohort Two (accrued in 2010-201 1). The per classroom and per student rates may vary in future interventions and are intended as a 
guide rather than a prediction. 

Rounding may cause slight discrepancies in sums and differences. 

“Cost per student is calculated based on the average number of children in Head Start classrooms across the United States. Each classroom has 
between 17 and 20 students. Here the number used is 19, so in a smaller classroom, the cost per student would be higher. 

'’Trainer supervision was counted separately for Incredible Years and Preschool PATHS. Tools of the Mind included trainer supervision time in the 
trainers’ fees. 


confirmed by invoices from developers; in other cases, the costs were calculated using per 
classroom or per training rates provided by the developers. 

As can be seen in the table, program costs varied across enhancements. These differ- 
ences were affected by characteristics specific to each enhancement. For example, the number 
of training days varied across enhancements, as shown in Table 3.1, which affected the cost of 
trainer, teacher, and coach participation. Trainer fees and material costs also varied across 
enhancements, affecting the overall cost as well. As mentioned above, MDRC’s technical 
assistance and monitoring costs for Head Start CARES implementation are excluded from the 
table. ' 


Components of Teacher Training Costs 

As described in more detail below, training costs included trainer fees for training and class- 
room visits, trainer travel, teacher payments for attending training, the cost of training materials, 
and coach payments for attending training. 

Trainer Fees 

Trainers were compensated for their time spent at training sessions and visiting class- 
rooms. Trainer fees varied by enhancement: trainers were compensated based on a daily rate for 
Preschool PATHS and The Incredible Years, whereas for Tools of the Mind they were compen- 
sated based on a flat fee per classroom for training and for supervising coaches’ classroom 
visits. 


Trainer Travel 

The cost includes a per visit rather than per day training rate for trainer travel, based on 
a flat rate for travel and per diem expenses across all enhancements. As shown in Table 3.1, 
Preschool PATHS and Tools of the Mind scheduled the first two training sessions together so 
that trainer travel costs were reduced, while Incredible Years training sessions were held 
individually over the course of the year, requiring separate trainer travel to each training. The 
Incredible Years and Preschool PATHS required only one trainer per session, while Tools of the 
Mind required two trainers per session. Classroom visits for all three enhancements were 
generally scheduled either directly before or after each training session, so additional costs for 
classroom visits include the costs for hotels and daily expenses, as well as the trainer fee for 


'Technical assistance costs included site visits, a management information system (MIS) used for monitor- 
ing, support for training logistics, support for hiring coaches, developer check-ins, and cross-site coach phone 
calls. For a fuller description of technical assistance provided in Head Start CARES, see Chapters 3 and 4, and 
Appendix B, which describes the MIS. 
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those days. The Incredible Years included four days of classroom visits per coach; Preschool 
PATHS included two to three days of classroom visits; and Tools of the Mind visits were 
included in the flat Tools rate. 

Teacher Payments 

Lead and assistant teachers were both compensated for the time they spent attending 
training. The Head Start CARES demonstration provided grantees with site payments that were 
sufficient to cover compensation for lead and assistant teachers’ attendance at 100 percent of the 
training sessions. Teachers were paid when training was on a weekend or another unpaid day. 
Grantees also used these payments to hire substitutes when the teachers were in training during 
a school day. 

Training Materials 

Each set of teachers from a classroom participating in the program group received train- 
ing materials that varied in cost across the enhancements. Training materials included items 
such as manuals and manipulatives (physical objects that children use to learn concepts). The 
Incredible Years provided a book and workbook for every teacher. Preschool PATHS provided 
each classroom with a set of classroom materials, such as puppets and books, and each teacher 
in the classroom received a set of manuals. Tools of the Mind manuals were shared by the lead 
and assistant teacher in each classroom. In addition, for Cohort One, it became clear that some 
classrooms or centers were lacking in basic materials that were needed to implement the 
enhancement. To supplement the materials provided to classrooms, the training materials costs 
for Cohort Two also include classroom gift cards and funds for administrative support to each 
grantee. 


Coach Payments 

Coaches attended all training sessions along with their teachers. Coaches were compen- 
sated at an hourly rate for their time spent at training. The typical rate for coaches, about $25 per 
hour, was used to estimate the costs in Appendix Table D.l. 

While the table shows average costs per classroom and per student for each enhance- 
ment, it is worth noting that there was significant variation in implementation costs across 
grantees based on region, grantee size, and many other factors. Additionally, because of the 
nature of the research study, grantees were required to offer training for three different en- 
hancements, which probably made training more costly than it would have been if only one 
enhancement had been implemented per grantee. Finally, incidental costs such as training space 
and equipment rentals for the training sessions varied widely by grantee and are not included in 
these estimates. 
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Components of Coaching Costs 

Coaches were hired to spend 90 minutes in each classroom per week. Coaches were either full- 
or part-time employees of the grantee and most were hired at the start of the implementation 
year by the grantee. Coaches were supervised by enhancement trainers and were provided 
minor administrative support by grantee liaisons, whose costs are not included here. The 
number of coaches also varied across enhancements. For more information about other aspects 
of the coaching process, see an earlier report that focused specifically on coaching in Head Start 
CARES. ^ As described below, coaching costs included coach salaries, coach training, and 
trainer supervision of coaches. 

Coach Salaries 

Coach salaries included observation and meeting time with teachers, time visiting class- 
rooms with trainers, supervision time with trainers, and time for general preparation and 
reporting. As described above, coaches were paid an average hourly rate of $25. Coaches’ 
overall salaries varied across enhancements, as each enhancement had a different number of 
training days across the year. 

Coach Training 

Coaches were trained for one day before the start of the year for Cohort One by trainers 
and developers for their respective enhancements. For Cohort Two, all coaches gathered in a 
central location and were trained for three days in advance of the start of the school year; they 
also spent three additional days of preparation time before working with the teachers. 

Coach Supervision 

Coaches were supervised by trainers, with check-ins scheduled at least once a month. 
Trainers and coaches were compensated for this time. 


Cost Estimates 

Costs are calculated for one year of implementation. All costs that are presented in Appendix 
Table D.l combine those from Cohort One (accmed in 2009-2010) and Cohort Two (accmed in 
2010-2011). Costs are calculated per classroom and per student, assuming 19 students per 
classroom, roughly the average for Head Start classrooms. Costs are averaged across grantees 
and therefore may reflect an average of costs between Cohort One and Cohort Two. 


^Lloyd and Modlin (2012). 
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Per classroom, implementing The Incredible Years cost about $10,990, while the cost 
per student was about $580. Incredible Years baining included six days of teacher training 
across the year, with a total cost comprising trainer fees, teacher compensation, and coach 
payments (to participate in training). Incredible Years coaching costs also included additional 
trainer time compared with other enhancements, based on the Incredible Years supervision 
model, as those trainers spent a full day after every training sessions supporting coaches in 
classrooms. The number of training visits is based on the estimated number of visits (four visits 
a year per coach), not the actual number of visits. 

Per classroom, implementing Preschool PATHS cost about $8,810, while the cost per 
student was about $460. PATHS included four days of training across the year, which, along 
with lower trainer fees per day than the other enhancements, lowered the overall cost. 

Tools of the Mind cost about $10,480 per classroom, while the cost per student was 
about $550. Tools included five days of training across the year and an additional $300 per 
classroom for supplemental materials, such as white boards or play props. Tools training 
sessions were also led by two trainers. Incredible Years and PATHS training sessions were led 
by one trainer each. 
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Appendix E 

Components of the Head Start CARES Enhancements 
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This appendix outlines the distinct and overlapping foci of the three social-emotional enhance- 
ments implemented in the Head Start CARES demonstration: The Incredible Years, Preschool 
PATHS, and Tools of the Mind — Play.' 

A number of differing interventions aimed at improving preschool children’s social- 
emotional development were considered for inclusion in the demonstration. The process that 
was used to identify interventions involved three phases. First, a list of candidate interventions 
was reviewed, using a set of criteria to determine whether these interventions were ready to “go 
to scale” in a national effectiveness trial. Second, the research base was reviewed for the small 
number of interventions that satisfied the first criterion. Finally, the content and organization of 
the identified candidates were reviewed to determine the similarity and differences between 
interventions. 

From the review of existing interventions, three interventions were identified as being 
ready for scale and having a research base. The three interventions were each selected to 
represent a different approach to enhancing social-emotional development: (1) an implicit 
approach to enhancing social-emotional development via effective classroom management (The 
Incredible Years); (2) an explicit, instmctional approach (Preschool PATHS); and (3) a scaf- 
folded play approach using peer-mediated play and learning structures to foster social- 
emotional development and self-regulation skills (Tools of the Mind). 

In the final phase of the selection process, the content and organization of the identified 
candidate interventions were examined. The purpose of this final phase of review was to 
compare the prototype interventions representing each of the three approaches to enhancing 
social-emotional development to determine whether they were well discriminated from each 
other, based upon clear differences in the content and organization of the intervention training 
and implementation manuals. In this part of the review, the stmcture of training and profession- 
al development support, as well as the demand on teacher and class time for implementation, 
were also examined. 

Notably, in this review, the focus was on those aspects of the training and implementa- 
tion manuals that were listed explicitly in the written materials gathered on each of the en- 
hancements. There was no attempt to gather further information from the developers on the 
more implicit information imparted in training or to observe training sessions or enhancement 
implementation to understand whether the written materials provided a comprehensive picture 
of each enhancement. There are two key advantages of such an approach: first, it allowed for a 
focus on those aspects of the enhancements that were so central that they were described in the 
written materials; second, more than one reviewer could check the material that was coded, 

'This appendix draws from a document developed at the start of Head Start CARES by Karen Bierman, 
Pamela Morris, Emily Snell, and Marcela Torres. 
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which reduced subjectivity. The disadvantage of this approach is that some aspects of the 
enhancements may have been missed in this coding. Indeed, several of the developers indicated 
that this approach did neglect to identify some aspects of their enhancements: The Incredible 
Years developers noted that the program also has a strong focus on the social and emotional 
skills of children; Preschool PATHS developers noted that their approach also supports teach- 
ers’ classroom management; and the developers of Tools of the Mind noted that social- 
emotional skills such as problem-solving or emotion identification are embedded in children’s 
learning through “pretend play” scenarios. 

Appendix Table E.l shows the results of this content- focused review (conducted before 
the start of the demonstration) and provides an overview of the unique foci of each enhance- 
ment as well as the crossover of enhancement emphases. Intervention emphasis was computed 
based on the number of hours/lessons out of the entire core intervention that were devoted to 
each domain of skills. The estimates for Tools of the Mind were based on the program’s first 
manual, which introduces the core approach to scaffolded play and other basic classroom 
learning structures. (Additional Tools manuals add learning structures and more advanced 
activities.) 

Interventions were examined module-by-module, and each lesson (or hour of teacher 
training) was assigned a primary target skill, based on its specific content. Modules that targeted 
multiple skill areas were assigned the skill area that was primarily targeted, such that each 
module was counted only once. The emphasis of each intervention by skill area is documented 
in the top portion of the table. The table clearly shows that the relative emphasis of each 
intervention discriminates between a focus on classroom management (The Incredible Years), 
social-emotional learning (Preschool PATHS), and restmcturing and scaffolding “pretend play” 
and learning activities (Tools of the Mind). 
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Head Start CARES Demonstration 

Appendix Table E.l 

Summary of Enhaneement Components 


Core Program Emphasis (%) 

Incredible Years Teacher Tools of the Mind: 

Training Program: Preschool PATHS: Scaffolded Play and Learning 

Program Emphasis and Component Classroom Management Social-Emotional Learning Activities 

Program emphasis 


Social-emotional learning (SEL) 

2.7 


88.6 

27.5 

Classroom management (CM) 
Deliberative/cooperative play skills 

80.3 


Embedded in lessons and extension 
activities 

Embedded in restructured 
classroom organization 

and communication (DPS) 

Not a core program 

emphasis 

9.1 

47.8 

Other (OTH) 

Child skills targeted** 

Emotional self-regulation/ 

16.6 



24.7 

behavioral inhibition (SEL) 
Emotional knowledge/ 

Embedded in CM 


15.9 

27.5 

expression empathy (SEL) 
Deliberative/cooperative play 

>2.7 


54.5 

Embedded in play sessions 

skills and communication (DPS) 

>2.7 


9.1 

47.8 

Social problem-solving (SEL) 

>2.7 


2.3 

Embedded in play sessions 

Self-esteem (OTH) 

Not a core program 

emphasis 

9.1 

Not a core program emphasis 

Planning/organization (DPS) 

Not a core program 

emphasis 

Not a core program emphasis 

9.8 

Literacy/math/science (OTH) 

Teacher skills targeted** 

Not a core program 

emphasis 

Not a core program emphasis 

14.9 

Positive behavior support (CM) 

8.3 


Embedded in lessons 

Embedded in restructured 
classroom organization 

Behavioral control strategies (CM) 

33.3 


Not a core program emphasis 

Not a core program emphasis 

Behavior management (CM) 

16.6 


Not a core program emphasis 

Not a core program emphasis 

Limit-setting (CM) 

16.6 


Not a core program emphasis 

Not a core program emphasis 

Teacher-child relationship (CM) 

5.5 


Embedded in lessons 

Embedded in restructured 
teacher-student interactions 

Classroom climate/structure (OTH) 

13.9 


9.1 

Embedded in restructured 
classroom organization 


SOURCE: Calculations drawn from content analysis of enhancement documents at the start of Head Start CARES by Karen Bierman, Pamela 
Morris, Emily Snell, and Marcela Torres. 

“The program emphasis for each skill is shown in parentheses, as defined in the first panel. 
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Appendix F 

Adapted Teaching Style Rating Scale (Adapted TSRS) 
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CLASSROOM STRUCTURE AND MANAGEMENT 

Consistency/Routine 

The teacher has the classroom operating smoothly and with a positive sense of 
teamwork. The teacher clearly states “mles and expectations” and his/her “routines 
are clearly understood by the classroom community.” 

Low Med High 

1 2 3 4 5 

Preparedness 

The teacher has clear plans, can deliver instructions and get children organized and 
engaged relatively quickly. The class mns as a “well-oiled machine.” 

1 2 3 4 5 

Classroom Awareness 

The teacher is aware of what is happening in all areas of the classroom by constant- 
ly scanning and monitoring. The teacher is regularly able to anticipate the individu- 
al needs of specific children while also attending to the class. For example, she may 
give instructions for children who finish task/meakactivity early, or is able to 
redirect child/children before conflict or dismptive episodes break(s) out. 

1 2 3 4 5 

DISCIPLINE 

Positive Behavior Management 

The teacher is successful in controlling children’s behavior, relying on positive 
behavior management strategies (rewarding good behavior with specific praise) and 
clear contingencies (reminding of mles, logical consequences). He/she regularly 
uses eye contact, simple verbal cues (e.g., clear directions, mentioning child by 
name), touch and other positive strategies to keep the class on track. 

1 2 3 4 5 

Negative Behavior Management 

The teacher may yell at children, make threats to withhold privileges, may display 
“many instances of using a tone of voice that conveys sarcasm and/or abmptness 
and/or harshness.” The teacher may be overly rigid or restrictive. 

1 2 3 4 5 

EMOTIONAL COMMUNICATION & SUPPORT 

Emotion Modeling 

The teacher frequently takes advantage of multiple opportunities to teach children 
about their emotions by identifying and labeling children’s emotional experiences. 
The teacher tells children about his/her own emotional experiences and models or 
demonstrates techniques to calm down (e.g., self-talk, deep breath). 

1 2 3 4 5 

Emotion Expression 

The teacher creates an environment that is supportive of children’s emotional 
expression. He/she encourages children to talk about their emotions and validates 
these experiences by providing verbal and physical support when children express 
themselves. 

1 2 3 4 5 

Emotion Regulation 

The teacher encourages and supports children to calm down using the strategies 
described, both when children are showing signs of becoming emotionally aroused 
but also preventativefy at times when it is usefril to do so (e.g., before coming inside 
from play). 

1 2 3 4 5 
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SOCIAL AWARENESS & PROBLEM SOLVING 

Social Awareness 

The teacher facilitates children’s empathy and perspective-taking skills by drawing 
attention to peers’ emotional experiences and explaining the interpersonal conse- 
quences of children’s actions. He/she encourages children to provide support to one 
another (e.g. “Ask your friend how he feels”, “What can you do to help your 
friend?”). 

1 2 3 4 5 

Social Problem Solving 

The teacher approaches problems between children as opportunities for learning. 
He/she provides appropriate support that matches children’s skill and language 
level, remains calm, and uses active listening to facilitate children’s communication 
and ability to collaborate in generating possible solutions. 

1 2 3 4 5 

PROMOTING SELF-REGULATION & PREVENTING MISBEHAVIOR 

Provision of Interpersonal Support 

The teacher uses verbally and/or physically supportive strategies (eye contact, 
physical contact, reassuring vocal tone, helping child to put feelings into words) to 
help children regain emotional control. This may include a teacher who notices and 
anticipates children’s escalating emotional distress, and who takes steps to support 
de-escalation. 

1 2 3 4 5 

Attention/Engagement 

The teacher uses several different gestures, cues, and signals to get and keep the 
attention of the entire class. 

1 2 3 4 5 

SCAFFOLDING 

Scaffolding Dramatic Play 

The teacher elicits children’s plans and helps to extend them by asking questions, 
etc. The teacher helps children to expand the story line, to extend story themes and 
actions. It does not include teacher participation in the play. 

1 2 3 4 5 

Scaffolding Peer Interaction 

The teacher elicits children’s plans, helping to extend them by asking questions, etc. 
The teacher helps children to expand collaborative play and to extend story themes 
and actions. It does not include teacher participation in the play. 

1 2 3 4 5 

Talk Aloud 

The teacher “talks aloud” as he/she plans or problem-solves with children. This 
includes stating his/her plans, talking about the challenge/task he/she’s trying to 
complete, and the solutions he/she could use to solve the problem or challenge. 

1 2 3 4 5 
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